Continuous Integration (CI) and Continuous Deployment (CD) are key practices in modern software development that streamline the release process. CI involves frequently integrating code changes into a central repository and running automated tests to catch issues early. CD, on the other hand, focuses on the automated release of code changes to production after they pass through the CI process.
We will explore the concept with sample example and easy to follow steps.
git clone <repository_url>
cd <repository_name>
git checkout -b ci-cd-setup
git add .
git commit -m "Add Python project files"
git push origin ci-cd-setup
ci-cd-setup branch into the main branch..github/workflows.workflows directory, create a YAML file (e.g., ci-cd.yml) for your CI/CD configuration. name: Python application CI, Build, Release, and Containerize
on:
push:
branches: ["main"]
pull_request:
branches: ["main"]
permissions:
contents: write # Need write permission to create releases/tags and upload artifacts
packages: write # Need write permission to push to GHCR
jobs:
# --- Build and Test Job ---
build_and_test:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: "pip"
- name: Install Test Dependencies
run: |
python -m pip install --upgrade pip
if [ -f requirements.txt ]; then
pip install -r requirements.txt
else
echo "Dependency file not found!"
exit 1
fi
- name: Test with pytest and check coverage
run: |
pytest --cov=src --cov=. --cov-report=term-missing --cov-report=xml --cov-fail-under=80
# --- Build and Push Docker Image Job ---
build_and_push_docker:
needs: build_and_test
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Log in to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: $
password: $
- name: Build Docker Image
run: |
docker build -t ghcr.io/$/$:latest .
- name: Push Docker Image
run: |
docker push ghcr.io/$/$:latest
- name: Output Docker Image Details
id: docker_meta
run: |
echo "image_tag=ghcr.io/$/$:latest" >> $GITHUB_ENV
echo "image_digest=$(docker inspect --format='' ghcr.io/$/$:latest)" >> $GITHUB_ENV
env:
GITHUB_ENV: $GITHUB_ENV
# --- Create Release Job ---
create_release:
needs: build_and_push_docker
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Create Release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: $
with:
tag_name: "v$"
release_name: "Release v$"
body: |
Automated release created by CI/CD workflow.
- Docker Image: ghcr.io/$/$:latest
draft: false
prerelease: false
# --- Deploy Job ---
deploy:
needs: build_and_push_docker
runs-on: self-hosted
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Deploy Code to Runner
run: |
mkdir -p /home/user/deployed_app
rsync -av --progress ./ /home/user/deployed_app --exclude .git --exclude .github --exclude .gitignore
nohup fastapi run --reload &
echo "Code deployed to /home/user/deployed_app"
build_and_test, build_and_push_docker, create_release and deploy. The build_and_test job installs dependencies and runs tests, build_and_push_docker job build docker image and push it to github container registry by tagging with gitrepo, create_release creates release with autogenerated version number and image details, and deploy deploys the application to production using github runner..github/workflows/ci-cd.yml file.
git add .github/workflows/ci-cd.yml
git commit -m "Add CI/CD configuration"
git push origin main
In this tutorial, we explored how to set up a CI/CD pipeline using GitHub Actions for a Python project. We created a GitHub repository, configured the CI/CD pipeline, and monitored its execution. By automating the build, test, and deployment processes, we can ensure that our code is always in a deployable state, leading to faster and more reliable software releases. Sample code along with the configuration file is provided for reference and can be accessed here
Thinking of learning Python or planing to change career track to be Python developer.
Let us start with few job profile where Python developers are in demand: Python Developer, Data Analyst, Automation Engineer, Full Stack Python Developer, Backend Engineer, Data Engineer, Machine Learning Engineer
All right! then let us start the journey of learning Python.
Go through the books listed here and based on your level of understanding, take any book and start reading it. In case you did not understand any topic or example of the book, don’t stop and spend too much time on it. You can always come back later and go through again.
Login to codewars.com and try some programming challege whenever you get sometime. It helps in mastering algorithm and python skill.
To be continued…
]]>In this post, we will continue our learning from basic to advance topics of python.
Interfacing with C/C++: * Use ctypes and cffi to interface with C libraries. * Explore writing Python extensions in C for performance-critical code. * Resources: * Ctypes * CFFI * Python Extensions in C
Deployment and Distribution: * Learn about packaging and distributing Python applications with setuptools and pip. * Explore containerization with Docker and deployment on cloud platforms. * Resources: * Setuptools * Docker * Deploying Python Applications
Basic Calculator:
Create a simple calculator that can perform basic arithmetic operations like addition, subtraction, multiplication, and division.
To-Do List:
Build a command-line or GUI-based to-do list application where users can add, remove, and mark tasks as completed.
Weather App:
Develop a weather application that fetches and displays weather information for a given location using an API like OpenWeatherMap.
Quiz Game:
Create a quiz game that asks users multiple-choice questions and keeps track of their score.
Currency Converter:
Build a currency converter that fetches real-time exchange rates and converts amounts between different currencies.
Simple Chatbot:
Develop a basic chatbot that can respond to user inputs with predefined responses.
Password Generator:
Create a password generator that generates strong, random passwords based on user-specified criteria (length, use of special characters, etc.).
Number Guessing Game:
Build a number guessing game where the computer randomly selects a number, and the user has to guess it within a certain number of attempts.
File Organizer:
Develop a script that organizes files in a directory based on their file types (e.g., move all images to an “Images” folder).
Expense Tracker:
Create an expense tracker that allows users to log their expenses and view summaries of their spending.
Contact Book:
Build a contact book application where users can add, remove, and search for contacts.
Simple Web Scraper:
Develop a web scraper that extracts information from a website and saves it to a file.
BMI Calculator:
Create a Body Mass Index (BMI) calculator that takes a user’s weight and height and calculates their BMI.
Rock, Paper, Scissors Game:
Develop a simple Rock, Paper, Scissors game where the user plays against the computer.
Mad Libs Game:
Create a Mad Libs game where the user inputs words, and the program generates a funny story using those words.
Hangman Game:
Build a Hangman game where the user guesses letters to figure out a hidden word.
Tic-Tac-Toe Game:
Develop a Tic-Tac-Toe game that can be played between two players or against the computer.
Flashcard App:
Create a flashcard application for studying vocabulary or other subjects.
Unit Converter:
Build a unit converter that converts between different units of measurement (e.g., length, weight, temperature).
Simple Alarm Clock:
Develop a simple alarm clock that allows users to set an alarm and notifies them when the time is reached.
Downloading file from internet can be time consuming work, especially when file to be downloaded are large files. During my project work i come across a situation where i had to download some files from internet and unzip it. Using request library with normal approach it was taking hours. After lot of search i found an arcticle which helped me downloading file faster.
import asyncio
import os.path
import shutil
from random import randint
from time import sleep
import random
import time
import aiofiles
import aiohttp
from urllib.parse import urlparse
from tempfile import TemporaryDirectory
# File Download class
class FileDownloader:
def __init__(self, urls, filename, download_path):
self.urls = urls
self.filename = filename
self.download_path = download_path
self.fast_download = None
async def get_content_length(self, url):
async with aiohttp.ClientSession() as session:
async with session.head(url) as request:
return request.content_length
def parts_generator(self, size, start=0, part_size=50 * 1024 ** 2):
while size - start > part_size:
yield start, start + part_size
start += part_size
yield start, size
async def download(self, url, headers, save_path, attempt=0):
# attempt = 0
max_attempts = 10
timeout = aiohttp.ClientTimeout(total=6000)
try:
async with aiohttp.ClientSession(headers=headers, timeout=timeout) as session:
await asyncio.sleep(1 + random.randint(0, 9))
async with session.get(url) as request:
file = await aiofiles.open(save_path, 'wb')
await file.write(await request.content.read())
except (aiohttp.ClientOSError,
aiohttp.ServerDisconnectedError):
if attempt < max_attempts:
print(f'Attempt {attempt} failed, trying again')
await asyncio.sleep(3 + random.randint(0, 9))
await self.download(url, headers, save_path, attempt + 1)
else:
raise
async def process(self, url):
filename = os.path.basename(urlparse(url).path)
# tmp_dir = TemporaryDirectory(prefix=filename, dir=os.path.abspath('.'), ignore_cleanup_errors=False)
tmp_dir = TemporaryDirectory(prefix=filename, dir=self.download_path)
size = await self.get_content_length(url)
tasks = []
file_parts = []
for number, sizes in enumerate(self.parts_generator(size)):
part_file_name = os.path.join(tmp_dir.name, f'{filename}.part{number}')
file_parts.append(part_file_name)
tasks.append(self.download(url, {'Range': f'bytes={sizes[0]}-{sizes[1]-1}'}, part_file_name))
await asyncio.gather(*tasks)
download_filename = os.path.join(self.download_path, filename)
# delete if there is existing zip file
if os.path.exists(download_filename):
os.remove(download_filename)
with open(download_filename, 'wb') as wfd:
for f in file_parts:
with open(f, 'rb') as fd:
shutil.copyfileobj(fd, wfd)
def download_file(self):
print('slow download')
download_file = os.path.join(self.download_path, self.filename)
with requests.get(self.urls[0], stream=True) as r:
r.raise_for_status()
with open(download_file, 'wb') as f:
for chunk in r.iter_content(chunk_size=8192):
f.write(chunk)
async def main(self, fast_download=True):
if fast_download:
await asyncio.gather(*[self.process(url) for url in self.urls])
else:
self.download_file()
if __name__ == '__main__':
url = "<file url>"
start_code = time.monotonic()
loop = asyncio.get_event_loop()
fileDownloader = FileDownloader(urls=[url], filename='file.txt.gz', download_path='/tmp')
loop.run_until_complete(fileDownloader.main())
print(f'{time.monotonic() - start_code} seconds!')
import os
import gzip
import time
# file unzip class
class FileUnzip:
def __init__(self, filename, file_path, unzip_path):
self.filename = filename
self.file_path = file_path
self.unzip_path = unzip_path
self.chunk_size = 100*1024*1024 # 100MB
def unzip(self):
zip_file = os.path.join(self.file_path, self.filename)
unzip_file = os.path.join(self.unzip_path, self.filename.replace('.gz', ''))
# delete if there is existing zip file
if os.path.exists(unzip_file):
os.remove(unzip_file)
with gzip.open(zip_file, 'rb') as f_in:
with open(unzip_file, 'wb') as f_out:
chunk = f_in.read(self.chunk_size)
while chunk:
f_out.write(chunk)
chunk = f_in.read(self.chunk_size)
if __name__ == '__main__':
start_code = time.monotonic()
fileUnzip = FileUnzip(filename='file.txt.gz', file_path='/tmp', unzip_path='/tmp')
fileUnzip.unzip()
print(f'{time.monotonic() - start_code} seconds!')
Redshift database is a cloud based, data warehouse solution offered by Amazon. It provides storage system that stores petabytes of data in easy-to-access clusters. Redshift is based on Postgres 8.0.2 and it supports sql syntax. Support for sql like query language makes it easy to analyze data for different kind of use cases.
Pentaho Data Integration eliminates coding and complexity, and puts data from any source at the fingertips of business users.
For debugging purpose, pip displays some installation message on console. It also offers ways to control console level log by -v,–verbose, -q and –quiet. Full log can be saved by providing –log option with file path, it will have complete installation log. There are cases where currupt download also creates issues at the time of installation, in that case try –no-cache-dir. If that also does not work then you can always try the installation from source file.
mylist=[1,2,3,'4','sample']
mydict = {1:'Python'}
f = open('<file name>','r')
print(f.read(10))
There are 2 types of membership operators in Python:
in: If the value is found in a sequence, then the result becomes true else false
not in: If the value is not found in a sequence, then the result becomes true else false
a=15
b=30
list = [3,6,15,20,25]
if (a in list):
print('a is available in given list')
else:
print('a is not available in given list')
if ( b not in list ):
print('b is not available in given list')
else:
print('b is available in given list')
mydict = {1:'Raj',2:'Som'}
print(mydict.keys())
Monkey patching means adding a new variable or method to a class or module after it’s been defined.
Example: We define a class A as
class A(object):
def __init__(self, num):
self.num = num
def __add__(self, other):
return A(self.num + other.num)
But now we want to add another function later in the code like below.
def get_num(self):
return self.num
To do this as method of class A at runtime, we assign the function to class like below.
A.get_num = get_num
At runtime, assign the new method and use new methos like below.
foo = A(42)
A.get_num = get_num
bar = A(6);
foo.get_num() # 42
bar.get_num() # 6
The special syntax, args and *kwargs in function definitions is used to pass a variable number of arguments to a function. The single asterisk form (*args) is used to pass a non-keyworded, variable-length argument list, and the double asterisk form is used to pass a keyworded, variable-length argument list.
*args example :
def test_var_args(farg, *args):
print('formal arg:{}'.format(farg))
for arg in args:
print('another arg:{}'.format(arg)
test_var_args(1, "two", 3)
Output
formal arg: 1
another arg: two
another arg: 3
**kwargs example
def test_var_kwargs(farg, **kwargs):
print('formal arg:{}'.format(farg))
for key in kwargs:
print('another keyword arg: {}: {}'.format(key, kwargs[key]))
test_var_kwargs(farg=1, myarg2="two", myarg3=3)
formal arg: 1
another keyword arg: myarg2: two
another keyword arg: myarg3: 3
When you create a list, you can read its items one by one. Reading its items one by one is called iteration:
mylist = [1, 2, 3]
for i in mylist:
print(i)
Output
1
2
3
mylist is an iterable. When you use a list comprehension, you create a list, and so an iterable:
mylist = [x*x for x in range(3)]
for i in mylist:
print(i)
Output
0
1
4
Everything you can use “for… in…” on is an iterable; lists, strings, files…
These iterables are handy because you can read them as much as you wish, but you store all the values in memory and this is not always what you want when you have a lot of values.
Generators are iterators, a kind of iterable you can only iterate over once. Generators do not store all the values in memory, they generate the values on the fly:
mygenerator = (x*x for x in range(3))
for i in mygenerator:
print(i)
Output
0
1
4
It is just the same except you used () instead of []. BUT, you cannot perform for i in mygenerator a second time since generators can only be used once: they calculate 0, then forget about it and calculate 1, and end calculating 4, one by one.
yield is a keyword that is used like return, except the function will return a generator.
Example:
def createGenerator():
mylist = range(3)
for i in mylist:
yield i*i
# create a generator
mygenerator = createGenerator()
# print generator object
print(mygenerator)
# print value
for i in mygenerator:
print(i)
Output
0
1
4
This is a simple example, but it’s handy when you know your function will return a huge set of values that you will only need to read once. To master yield, you must understand that when you call the function, the code you have written in the function body does not run. The function only returns the generator object. Then, your code will continue from where it left off each time for uses the generator.
The first time the for calls the generator object created from your function, it will run the code in your function from the beginning until it hits yield, then it’ll return the first value of the loop. Then, each other call will run the loop you have written in the function one more time, and return the next value until there is no value to return.
The generator is considered empty once the function runs, but does not hit yield anymore. It can be because the loop had come to an end, or because you do not satisfy an “if/else” anymore.
Like any other language classes in Python are also a piece of code that descibe how to produce an object. That is true for python as well.
class ObjectCreator(object):
pass
my_object = ObjectCreator()
print(my_object)
Output
<__main__.ObjectCreator object at 0x7f3d86c186d0>
But classes are more than that in Python. Classes are objects too. As soon as you use the keyword class, Python executes it and creates an OBJECT. The instruction
class ObjectCreator(object):
pass
creates in memory an object with the name “ObjectCreator”.
This object (the class) is itself capable of creating objects (the instances), and this is why it’s a class.
But still, it’s an object, and therefore:
print(ObjectCreator)
def echo(o): print(o)
echo(ObjectCreator)
print(hasattr(ObjectCreator, ‘new_attribute’))
ObjectCreator.new_attribute = ‘foo’
print(hasattr(ObjectCreator, ‘new_attribute’))
print(ObjectCreator.new_attribute)
ObjectCreatorMirror = ObjectCreator print(ObjectCreatorMirror.new_attribute)
print(ObjectCreatorMirror())
Output
```python
<class '__main__.ObjectCreator'>
<class '__main__.ObjectCreator'>
False
True
foo
foo
<__main__.ObjectCreator object at 0x7f455006b810>
Since classes are objects, you can create them on the fly, like any object. But it’s not so dynamic, since you still have to write the whole class yourself. For more details, Please refer link.
Metaclasses are the ‘stuff’ that creates classes, i.e. metaclasses are what create these objects. They are the classes’ classes.
MyClass = MetaClass()
my_object = MyClass()
type is the metaclass Python uses to create all classes behind the scenes.
Everything is an object in Python. That includes ints, strings, functions and classes. All of them are objects. And all of them have been created from a class:
>>> age = 35
>>> age.__class__
<type 'int'>
>>> name = 'bob'
>>> name.__class__
<type 'str'>
>>> def foo(): pass
>>> foo.__class__
<type 'function'>
>>> class Bar(object): pass
>>> b = Bar()
>>> b.__class__
<class '__main__.Bar'>
Now, what is the class of any class ?
>>> age.__class__.__class__
<type 'type'>
>>> name.__class__.__class__
<type 'type'>
>>> foo.__class__.__class__
<type 'type'>
>>> b.__class__.__class__
<type 'type'>
type is the built-in metaclass Python uses, but there is a way to create own metaclasses by defining metaclass attribute.
Python 2 syntax for creating metaclass:
class Foo(object):
__metaclass__ = something...
[...]
Python 3 syntax for creating metaclass:
class Foo(object, metaclass=something):
...
You write class Foo(object) first, but the class object Foo is not created in memory yet.
Python will look for metaclass in the class definition. If it finds it, it will use it to create the object class Foo. If it doesn’t, it will use type to create the class.
For detail explaination please refer stackoverflow link.
In Python 3.5 and above to mearge two dictionaries x and y below syntax is used.
z = {**x, **y}
In Python 2, (or 3.4 or lower) write a function:
def merge_two_dicts(x, y):
z = x.copy() # start with x's keys and values
z.update(y) # modifies z with y's keys and values & returns None
return z
z = merge_two_dicts(x, y)
For dictionaries x and y, z becomes a shallowly merged dictionary with values from y replacing those from x.
print('.', end='')
Decorators are functions which modify the functionality of other functions. In another word decorators wrap a function, modifying its behavior. Let us start with simple example:
def my_decorator(func):
def wrapper():
print("Something is happening before the function is called.")
func()
print("Something is happening after the function is called.")
return wrapper
def say_whee():
print("Whee!")
say_whee = my_decorator(say_whee)
say_whee()
Output
Something is happening before the function is called.
Whee!
Something is happening after the function is called.
In the above code, decoration happened at line.
say_whee = my_decorator(say_whee)
The way you decorated say_whee() above is a little clunky.Python allows you to use decorators in a simpler way with the @ symbol.
Same code can be re-written.
def my_decorator(func):
def wrapper():
print("Something is happening before the function is called.")
func()
print("Something is happening after the function is called.")
return wrapper
@my_decorator
def say_whee():
print("Whee!")
Output
Something is happening before the function is called.
Whee!
Something is happening after the function is called.
For more details, Please refer the link.
Whenever the Python interpreter reads a source file, it does two things:
>>> class MyClass():
... def __init__(self):
... self.__superprivate = "Hello"
... self._semiprivate = ", world!"
...
>>> mc = MyClass()
>>> print mc.__superprivate
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: myClass instance has no attribute '__superprivate'
>>> print mc._semiprivate
, world!
>>> print mc.__dict__
{'_MyClass__superprivate': 'Hello', '_semiprivate': ', world!'}
NumPy is the fundamental package for scientific computing in Python. It is a Python library that provides a multidimensional array object, various derived objects (such as masked arrays and matrices), and an assortment of routines for fast operations on arrays, including mathematical, logical, shape manipulation, sorting, selecting, I/O, discrete Fourier transforms, basic linear algebra, basic statistical operations, random simulation and much more. For more details please refer numpy.
pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python.
pandas is well suited for many different kinds of data:
For getting started with basic pandas, Please refer 10 minutes to pandas
Using DataFrame.iterrows (generator) which yields both index and row.
import pandas as pd
df = pd.DataFrame([{'c1':10, 'c2':100}, {'c1':11,'c2':110}, {'c1':12,'c2':120}])
for index, row in df.iterrows():
print(row['c1'], row['c2'])
Output:
10 100
11 110
12 120
Data files which does not fit into memory can be ready in chunk sizes. Pandas has option to define chunk size while reading the data file.
for chunk in reader: print(chunk)
* Option 2: Specify iterator=True to read_csv, it also return an iterable object of type TextFileReader.
```sql
reader = pd.read_csv('tmp.sv', sep='|', iterator=True)
reader.get_chunk(5)
Instance method: Method associated with instance, such method first parameter is self. It is used for handling instance related variables.
Class method: Method associate with class and declared by using classmethod decorator. It is used for handling class related variables.
Static method: Method that is part of class but not related to instance or class variable, it is declared using staticmethod decorator.
import datetime
class Employee:
# class variable
num_of_emps = 0
raise_amt = 1.04
# instantiate employee class
def __init__(self, first, last, pay):
self.first = first
self.last = last
self.pay = pay
Employee.num_of_emps += 1
# instance method
def fullname(self):
return '{} {}'.format(self.first, self.last)
# instance method
def apply_raise(self):
self.pay = int(self.pay * self.raise_amt)
# instance method
def apply_raise2(self, amt):
self.pay = int(self.pay * amt)
# class method
@classmethod
def set_raise_amt(cls, amount):
cls.raise_amt = amount
# class method
@classmethod
def from_string(cls, emp_str):
first, last, pay = emp_str.split('-')
return cls(first, last, pay)
# static method
@staticmethod
def is_workday(day):
if day.weekday() == 5 or day.weekday() == 6:
return False
return True
# create employee class instance with value
emp_1 = Employee('Raj', 'Nath', 100)
emp_2 = Employee('Som', 'Nath', 200)
# print employee 1 name
print(emp_1.fullname())
print(emp_1.pay)
# print employee 2 name
print(emp_2.fullname())
print(emp_2.pay)
# raise employee 2 pay by 10% using instance method
emp_2.apply_raise2(1.10)
# print employee pay
print(emp_1.pay)
print(emp_2.pay)
# set employee raise pay to 5% using class method
Employee.set_raise_amt(1.05)
print(Employee.raise_amt)
print(emp_1.raise_amt)
print(emp_2.raise_amt)
emp_str_3 = 'Sonia-Dhawan-300'
emp_3 = Employee.from_string(emp_str_3)
print(emp_3.first)
print(emp_3.pay)
my_date = datetime.date(2016, 7, 11)
print(Employee.is_workday(my_date))
List is mutable, i.e. list can be modified after declaring it.
mylist =[1,2,3]
print(mylist)
# prints [1,2,3]
mylist.append(4)
print(mylist)
# prints [1,2,3,4]
Tuple is immutable, i.e. tuple can not be modified after declaring it.
mytuple = (1,2,3)
print(mytuple)
Module : File containing Python definition and statements. For more details refer module.
Package : Packages are way of structuring Python’s module namespace by using “dotted module name”. For more details refer package.
By defining variable in config.py file and importing it accross different modules.
range function return a range object.
r = range(1,3)
print(r)
# print range(1, 4)
print(type(r))
# print <class 'range'>
Using os.walk, all directory, sub directory and files can be listed.
import os
rootdir = os.getcwd()
for folder, subs, files in os.walk(rootdir):
print(files)
deep copy and shallow copy can be created using copy class method. Reference copy can be created by assign reference to another variable.
import copy
lst = [1,2,3]
print('List: {}'.format(lst))
# deepcopy which will create another copy
d_copy = copy.deepcopy(lst)
print('New copy: {}'.format(d_copy))
# any operation on this copy will not change original list
d_copy[1] = 123
print('After update (new_copy): {}'.format(d_copy))
print('Original list: {}'.format(lst))
#shallow copy
s_copy = copy.copy(lst)
print('After shallow copy: {}'.format(s_copy))
# operation on this copy data will not change original
s_copy.append(123)
print('Shallo copy after update: {}'.format(s_copy))
print('Original list : {}'.format(lst))
# reference copy
r_copy = lst
print('reference copy: {}'.format(r_copy))
r_copy[1]=123
print('reference copy: {}'.format(r_copy))
print('original copy: {}'.format(lst))
Output
List: [1, 2, 3]
New copy: [1, 2, 3]
After update (new_copy): [1, 123, 3]
Original list: [1, 2, 3]
After shallow copy: [1, 2, 3]
Shallo copy after update: [1, 2, 3, 123]
Original list : [1, 2, 3]
reference copy: [1, 2, 3]
reference copy: [1, 123, 3]
original copy: [1, 123, 3]
Operation can be performed on list and tuple, but it behaves differently when we try normally. In below example when multiplied by 2, it prints same list 2 times, which is not intended. In order to get intended result, try list comprehension or other option such map function.
# operation of list
lst = [1,2,3,4] # list
tpl = (1,2,3,4) # tuple
lm = lambda x : x*2 # lambda function
print('Original list: {}'.format(lst))
print('After multiplication with 2: {}'.format(lst*2))
print('Multiplication using list comprehension')
print([i*2 for i in lst])
print('Original list: {}'.format(lst))
print('Multiplication using list comprehension with lmbda function')
print([lm(x) for x in lst])
for i in map(lm,lst):
print(i)
# operation over tuple
print('Original tuple: {}'.format(tpl))
print('After muplication with 2: {}'.format(tpl*2))
print('Multiplication using list comprehension')
print([i*2 for i in tpl])
print('Multiplication using list comprehension with lmbda function')
for i in (lm(x) for x in tpl):
print(i)
Output
Original list: [1, 2, 3, 4]
After multiplication with 2: [1, 2, 3, 4, 1, 2, 3, 4]
Multiplication using list comprehension
[2, 4, 6, 8]
Original list: [1, 2, 3, 4]
Multiplication using list comprehension with lmbda function
[2, 4, 6, 8]
2
4
6
8
Original tuple: (1, 2, 3, 4)
After muplication with 2: (1, 2, 3, 4, 1, 2, 3, 4)
Multiplication using list comprehension
[2, 4, 6, 8]
Multiplication using list comprehension with lmbda function
2
4
6
8
Adding below line as first link in python script, makes it executable from anywhere.
#!/usr/bin/env python3
Let us take a case of YouTube, a user logged in and started watching video and watched from 0-10 min, later logged in again and watched same video from 15-20 min and 18-25 min. In this case total duration of viewer is 10+6+8 = 24 min but if we remove the timing where user watched same content, then total duration would be 10+6+5 = 21 min.
# Input
input_list = [(1, 10), (10,16)]
# Output
output_list = []
for lst in input_list:
output_list = output_list + list(range(lst[0], lst[1]))
# Total duration after removing duplicates
total_duration = len(list(set(output_list)))
# Total time
print(total_duration)
# Input data
str1 = 'raR'
if str1.lower() == str1[::-1].lower():
print('Palindrome.')
else:
print('Not a Palindrome.')
Context manager is used to manage resources correctly. Using context manager one can eliminate the chances of keeping resources such a file and database connections opened when it is no longer needed. Although as programmer it is expected to write code, taking care of releasing resources once that is no longer needed. Using context manager, it can be automated and resources can be released automatically. Most common use of inbuilt context manager is the function to open file.
Let us understand with example by creating our own custom function for handling file.
from contextlib import contextmanager
@contextmanager
def open_file(file_name, mode):
try:
f = open(file_name, mode)
yield f
finally:
f.close()
with open_file('log_a.txt', 'w') as f:
f.append('Hello')
print(f.closed)
print(f.closed)
On running above code sniplets, it will open file and write a word and print status of file as False (file is not closed) and then when code is out of context, it print status as True (file is closed).
There is another option of using classes.
class File():
def __init__(self, filename, mode):
self.filename = filename
self.mode = mode
def __enter__(self):
self.open_file = open(self.filename, self.mode)
return self.open_file
def __exit__(self, *args):
self.open_file.close()
files = []
for _ in range(10000):
with File('foo.txt', 'w') as infile:
infile.write('foo')
files.append(infile)
print(f.closed)
Above also does same, when code is out of context, it print status of file as closed.
In case of distributed system where different systems are involved in executing different parts, we often execute piece of code on different server. In such cases if you want to return some complex data structure like dictionary or list on the fly which is really needed in case of small data sets, thing become complex with no details around it. I found very good peice of code online to solve this problem.
# Parse data for sending as argument
import urllib.parse
import json
dict1_j = urllib.parse.quote(json.dumps(dict1))
# Unparse at receiving end
import json
import urllib.parse
dict1 = json.loads(urllib.parse.unquote(sys.argv[1]))
#Password Generator
letters = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z']
numbers = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
symbols = ['!', '#', '$', '%', '&', '(', ')', '*', '+']
print("Welcome to the PyPassword Generator!")
nr_letters= int(input("How many letters would you like in your password?\n"))
nr_numbers = int(input(f"How many numbers would you like?\n"))
nr_symbols = int(input(f"How many symbols would you like?\n"))
Using python random module choices and shuffle method, random password can be generated easily.
password_list = random.choices(letters,k=nr_letters) + random.choices(symbols,k=nr_symbols) + random.choices(numbers,k=nr_numbers)
print(password_list)
random.shuffle(password_list)
password = ''.join(password_list)
print('Random generated password :{}'.format(new_password))
Python dictionary key can be of any data type but it can not have mutable object/data type as key. In other a dictionary can not have a list or dict as key.
def sum(n):
if n <= 1:
return n
else:
return n + sum(n-1)
Airflow is a platform created by the community to programmatically author, schedule and monitor workflows.
Airflow scheduler run continuously in background and generates lots of logs and it fills the file system. These logs needs to be clean up periodically to free space.
find $AIRFLOW_HOME/logs/scheduler -type f -mtime +5 -delete
from datetime import datetime, timedelta
from airflow.models import DAG
from airflow.operators.bash_operator import BashOperator
main_dag_id = 'scheduler_cleanup'
args = {
'owner': 'Airflow',
'start_date': datetime(2021, 7, 21),
'provide_context': True
}
with DAG(
main_dag_id,
catchup=False,
concurrency=4,
schedule_interval='@daily',
default_args=args) as dag:
clean_scheduler_logs = BashOperator(task_id='clean_scheduler_logs',
bash_command="find $AIRFLOW_HOME/logs/scheduler -type f -mtime +7 -delete")
clean_scheduler_logs
SSH or Secure Shell is a cryptographic network protocol for operating network services securely over an unsecured network.[1] Typical applications include remote command-line, login, and remote command execution, but any network service can be secured with SSH.
Python has many libraries for using SSH. I will discuss about the two libraries which i have used in my projects.
Paramiko is a Python (2.7, 3.4+) implementation of the SSHv2 protocol [1], providing both client and server functionality. While it leverages a Python C extension for low level cryptography (Cryptography), Paramiko itself is a pure Python interface around SSH networking concepts.
Let us look at example of paramiko.
import paramiko
hostname = 'localhost'
username = 'root'
password = 'password'
script_path = "/root"
command = "ls"
pem = None
# ssh client
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
# connect to ssh server
client.connect(hostname =hostname,
username=username,
password=password,
pkey=paramiko.RSAKey.from_private_key_file(pem) if pem else None)
# run command
_, stdout, stderr = client.exec_command('cd {} && {}'.format(script_path, command))
out = stdout.readlines()
err = stderr.readlines()
exit_code = stdout.channel.recv_exit_status()
client.close()
AsyncSSH is a Python package which provides an asynchronous client and server implementation of the SSHv2 protocol on top of the Python 3.6+ asyncio framework. Lets look into one simple example to run command using asynssh.
import asyncio, asyncssh, sys
hosts = 'localhost'
username = 'username'
password = 'password'
# create connection
async def run_client():
conn = await asyncio.wait_for(asyncssh.connect(hosts, username=username, password=password, known_hosts = None),10,)
return conn
# run command
async def run_command():
try:
conn = await run_client()
result = await conn.run('ls')
if result.exit_status == 0:
print(result.stdout, end='')
else:
print(result.stderr, end='', file=sys.stderr)
print('Program exited with status %d' % result.exit_status,
file=sys.stderr)
except Exception as ex:
print(ex)
async def program():
await asyncio.gather(run_command())
# Run our program until it is complete
try:
asyncio.get_event_loop().run_until_complete(program())
except (OSError, asyncssh.Error) as exc:
sys.exit('SSH connection failed: ' + str(exc))
Above examples works well in most cases and it worked for me also, but in coporate network where everything is behind firewall, these firewalls causes issue in case of long running commands/scripts. Here firewall means the firewall software which are deployed on corporate networks. These firewall monitors network connections and if there is a long running idle sessions, then it disconnects automatically. It is a security measure but it can cause problems for programs which takes longer and there is communication between client and server. It looks simple but debugging this issue is bit tidious. For me it tooks weeks to come to this conclusion and then i started looking for solution. SSh has option to keep pinging after certain time but unfortunately, i could not find it anywhere.
Solution to this problem is very simple, We need to keep session alive by doing something during the long running idle session. Either you return something or pass something but bit tedius to implement. But ssh has very nice option to send ping after given interval, this helps in keeping the session active and alive. We see an example using paramiko where using transport, keep alive interval can be set.
import paramiko
# Remote server details
hostname = '172.*.*.*'
port = 22
username = 'root'
password = 'password'
# script and path
script_path = "/root/raj/"
command = "sh sleeptest.sh"
# set 60 sec interval to ping remote
client = paramiko.Transport((hostname, port))
client.set_keepalive(60)
client.connect(username=username, password=password)
stdout_data = []
stderr_data = []
session = client.open_channel(kind='session')
session.exec_command('cd {} && {}'.format(script_path, command))
stdout = session.makefile('rb', -1)
stderr = session.makefile_stderr('rb', -1)
print('exit status: ', session.recv_exit_status())
out = stdout.readlines()
err = stderr.readlines()
session.close()
client.close()
import asyncio, asyncssh, sys
async def run_client():
conn = await asyncio.wait_for(asyncssh.connect('172.*.*.*', username='root', password='password', known_hosts = None,
keepalive_interval=600, keepalive_count_max=10000),10,)
return conn
async def run_command():
try:
conn = await run_client()
result = await conn.run('cd /root/raj && sh sleeptest.sh')
if result.exit_status == 0:
print(result.stdout, end='')
else:
print(result.stderr, end='', file=sys.stderr)
print('Program exited with status %d' % result.exit_status,
file=sys.stderr)
except Exception as ex:
print(ex)
async def program():
# Run both print method and wait for them to complete (passing in asyncState)
await asyncio.gather(run_command())
# Run our program until it is complete
loop = asyncio.get_event_loop()
loop.run_until_complete(program())
loop.close()
Hope it will be helpful to anyone new to these library. Do let me know if you find this article useful and feel free to reach me using any medium to discuss on anything.
Happy coding!
Graphviz facilitates the creation and rendering of graph descriptions in the DOT language of the Graphviz graph drawing software from Python.
sudo apt install graphviz
pip install graphviz
from graphviz import Digraph
dot = Digraph(comment='The Round Table')
dot.format = 'png'
dot.attr(rankdir='LR')
dot.attr('node', shape='circle')
# Sample data as dictionary
dict_data = {'test suite 1':[{'test case 1':[{'test 1':'pass'},{'test 2':'fail'}]}, {'test case 2':[{'test 22':'pass'},{'test 23':'fail'}]}]}
# Node
for d in dict_data:
dot.node(d, d)
for dd in dict_data[d]:
for ddd in dd:
dot.node(ddd, ddd)
for dddd in dd[ddd]:
for t in dddd:
if dddd[t] == 'pass':
dot.node(t, t, color='green')
else:
dot.node(t, t, color='red')
# Edge
for d in dict_data:
for dd in dict_data[d]:
for ddd in dd:
dot.edge(d, ddd)
for dddd in dd[ddd]:
for t in dddd:
dot.edge(ddd, t)
dot.render('round-table', view=True)