In this chapter, we’ll come across another common routine, before that, you’ll have to go look for the basic python knowledge, because we’re not gonna talk about them again.

OS

Note that folder names and file names are not case-sensitive on Windows and OS X, they are only case-sensitive on Linux.

And on Windows, we use backslashes \ as separators between folders to organize path. But on OS X and Linux, we use forward slashes / as their path separators.

1
import os, sys
1
2
3
4
5
os.path.join('Program Files', 'Listary', 'Images') 
## Windows
#'Program Files\\Listary\\Images'
## OS X and Linux
#'Program Files/Listary/Images'

os.getcwd() : Get Current working directory,

os.chdir(): Change working directory,

1
2
3
4
5
os.getcwd()
# 'H:\\cocofiles\\Python'
os.chdir('H:/dorofiles/jupyter') ## return FileNotFoundError if not exists
os.getcwd()
# 'H:\\dorofiles\\jupyter'

Introduction

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
os.listdir("H:\oceanfiles\V2Ray\doc")
# ['readme.md']
 
os.mkdir('Tool')    ## return error if dir already exists
os.mkdir('Tool\hello') ## recursive , return error if the 1st dir already exists
os.makedirs('Tool\hello', exist_ok=True) 

path = "/tmp/home/monthly/daily"
os.makedirs( path, 0o777 ) ## create recursive folders in root, mode is 777

dirs = os.listdir(path)
for file in dirs:
    print (file)
# .ipynb_checkpoints
# Fiddler.exe
# Help.url
# DELETEME.txt
# LICENSE.txt
# Tools
# EMPTYFOLDER
# Untitled.ipynb

os.open

flags can be,

flags description
_O_BINARY Opens the file in binary (untranslated) mode. (See fopen for a description of binary mode.)
_O_CREAT Creates a file and opens it for writing. Has no effect if the file specified by filename exists. The pmode argument is required when _O_CREAT is specified.
_O_APPEND Moves the file pointer to the end of the file before every write operation.
_O_CREAT | _O_TEMPORARY Creates a file as temporary; the file is deleted when the last file descriptor is closed. The pmode argument is required when _O_CREAT is specified.
_O_CREAT | _O_EXCL Returns an error value if a file specified by filename exists. Applies only when used with _O_CREAT.
_O_NOINHERIT Prevents creation of a shared file descriptor.
_O_RANDOM Specifies that caching is optimized for, but not restricted to, random access from disk.
_O_RDONLY Opens a file for reading only. Cannot be specified with _O_RDWR or _O_WRONLY.
_O_RDWR Opens a file for both reading and writing. Cannot be specified with _O_RDONLY or _O_WRONLY.
_O_SEQUENTIAL Specifies that caching is optimized for, but not restricted to, sequential access from disk.
_O_TEXT Opens a file in text (translated) mode. (For more information, see Text and Binary Mode File I/O and fopen.)
_O_TRUNC Opens a file and truncates it to zero length; the file must have write permission. Cannot be specified with _O_RDONLY. _O_TRUNC used with _O_CREAT opens an existing file or creates a file. Note: The _O_TRUNC flag destroys the contents of the specified file.
_O_WRONLY Opens a file for writing only. Cannot be specified with _O_RDONLY or _O_RDWR.
_O_U16TEXT Opens a file in Unicode UTF-16 mode.
_O_U8TEXT Opens a file in Unicode UTF-8 mode.
_O_WTEXT Opens a file in Unicode mode.

For full list, you can have a reference from Microsoft official docs : https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/open-wopen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
## create a foo.txt, and write something
fd = os.open( "foo.txt", os.O_RDWR|os.O_CREAT )
res = os.write(fd, b"This is test")
print(res)  ## print bytes
# 12
os.close( fd )

# toWrite = "This is test"
# os.write(fd, bytes(toWrite, 'UTF-8'))

os.rename("foo.txt", "foobar.txt")
os.remove("DELETEME.txt")
os.unlink("foobar.txt")
os.rmdir('EMPTYFOLDER') ## remove empty folder


for line in open("version"):
	print(line, end="")
# 6.1.4

os.path

Theos.path module provides functions that help us deal with the system path, and can check whether a given path is an absolute path.

1
2
3
4
5
6
7
8
os.path.abspath('.') 
# 'C:\\Users\\Administrator' 
os.path.abspath('.\\.conda') 
# 'C:\\Users\\Administrator\\.conda' 
os.path.isabs('.') 
#  False 
os.path.isabs(os.path.abspath('.')) 
# True 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
os.path.relpath('C:\\Windows', 'C:\\') 
# 'Windows' 
os.path.relpath('C:\\Windows', 'C:\\Users\\Administrator') 
# '..\\..\\Windows' 

path = 'C:\\Windows\\System32\\calc.exe' 
os.path.basename(path) 
# 'calc.exe' 
os.path.dirname(path) 
# 'C:\\Windows\\System32' 

path = 'C:\\Windows\\System32\\calc.exe' 
path.split(os.path.sep) 
## Windows
# ['C:', 'Windows', 'System32', 'calc.exe']
'/usr/bin'.split(os.path.sep) 
## OS X and Linux
# ['', 'usr', 'bin']

Check the path:

1
2
3
4
5
6
7
8
9
10
os.path.exists('C:\\Windows') 
# True
os.path.exists('Untitled.ipynb') 
# True
os.path.isdir('C:\\Windows\\System32') 
# True 
os.path.isfile('C:\\Windows\\System32') 
# False 
os.path.isfile('C:\\Windows\\System32\\calc.exe') 
# True

List files in folder and view file size :

1
2
3
4
5
6
7
8
9
10
import os
fileDict = dict()
totalSize = 0 
path = 'C:\\Users\\Administrator\\.ssh\\'
for f in os.listdir(path): ## Calculate attributes for all files listed here. 
    fileDict[f] =  os.path.getsize(os.path.join(path, f)) 
    totalSize = totalSize + fileDict[f]
for item in fileDict:
    print("The File: {:30s} The Size: {:d} Bytes".format(item, fileDict[item]))
print("Total Size： {:d} Bytes".format(totalSize))
1
2
3
4
The File: id_rsa                         The Size: 1679 Bytes
The File: id_rsa.pub                     The Size: 400 Bytes
The File: known_hosts                    The Size: 1382 Bytes
Total Size： 3461 Bytes

rename.py

1
2
3
4
5
6
7
8
9
10
import os
path = 'D:\\ST3'
i = 1
for file in os.listdir(path):
    if os.path.isfile(os.path.join(path, file)) == True:
        ## if the destination file exists

        new_name = file.replace(file, "%d_%s_encrypted" % (i, file.split('.')[0]))
        os.rename(os.path.join(path, file), os.path.join(path, new_name))
        i += 1

os.stat

1
2
3
4
5
6
7
8
9
import os
fileDict = dict()
path = 'C:/Users/Administrator/.ssh/'
for f in os.listdir(path): ## Calculate attributes for all files listed here. 
    fileStats = os.stat(path + f)
    fileDict[f] = fileStats
    print(fileStats)
for item in fileDict:
    print("The File: {:30s} The Size: {:d} Bytes".format(item, fileDict[item].st_size))
1
2
3
4
5
6
os.stat_result(st_mode=33206, st_ino=5348024557680855, st_dev=1651831854, st_nlink=1, st_uid=0, st_gid=0, st_size=1679, st_atime=1577672408, st_mtime=1577672408, st_ctime=1577672408)
os.stat_result(st_mode=33206, st_ino=13510798882289897, st_dev=1651831854, st_nlink=1, st_uid=0, st_gid=0, st_size=400, st_atime=1577672408, st_mtime=1577672408, st_ctime=1577672408)
os.stat_result(st_mode=33206, st_ino=27021597764384213, st_dev=1651831854, st_nlink=1, st_uid=0, st_gid=0, st_size=1382, st_atime=1577672622, st_mtime=1584688410, st_ctime=1577672622)
The File: id_rsa                         The Size: 1679 Bytes
The File: id_rsa.pub                     The Size: 400 Bytes
The File: known_hosts                    The Size: 1382 Bytes

Fiddler

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import os
import os.path

def getFile(filePath, pat="py", files=[]):
    fl = os.listdir(filePath)  ## list all files in path
    try:
        for f in fl:
            tmpath = os.path.join(filePath, f) #== path + '/' + f
            if os.path.isdir(tmpath):	## check if it is a dir
                getFile(tmpath, filePath, files)  ## recursive
            elif tmpath[tmpath.rfind('.') + 1:].lower() == pat:
                files.append(tmpath)  
            ## find that filename extension and add to the tmppath
        else:
            return files
    except PermissionError:
        print("Permission denied: '%s'" % filePath)
while True:
    path = input('Input Your Path Here:').strip()
    if os.path.isdir(path) == True:
        print("Path:", path)
        break
# path = 'D:/networkTools/Fiddler'
txtfiles = getFile(path, "txt")
print(txtfiles)
1
['D:\\networkTools\\Fiddler\\credits.txt', 'D:\\networkTools\\Fiddler\\license.txt']

os.access

Create a file named HIDEME.txt(readonly and automatically hide)

1
2
3
4
5
6
7
8
os.access("HIDEME.txt", os.F_OK) ## if exits
# True
os.access("HIDEME.txt", os.R_OK) ## if readable
# True
os.access("HIDEME.txt", os.W_OK) ## if writable
# True
os.access("HIDEME.txt", os.X_OK) ## if excutable
# False

os.system

os.system() method execute the command (a string) in a subshell.

1
2
3
4
5
import os  ## importing os module  
# dir(os)
os.system('mkdir py')  ## Command to execute and make a py directory
os.system('notepad')  ## open notepad.exe
os.system('notepad python.txt')

os.walk

os.listdir() only list files in current folder, but os.walk will walk through all files and subfolders in current folder.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
tree /f
C:.
│  jupyter_notebook_config.py
│  migrated
│
└─lab
    ├─user-settings
    │  └─@jupyterlab
    │      ├─apputils-extension
    │      │      themes.jupyterlab-settings
    │      │
    │      ├─codemirror-extension
    │      │      commands.jupyterlab-settings
    │      │
    │      ├─extensionmanager-extension
    │      │      plugin.jupyterlab-settings
    │      │
    │      └─filebrowser-extension
    │              browser.jupyterlab-settings
    │
    └─workspaces
            lab-a511.jupyterlab-workspace
            labworkspacesauto-2-f9e3.jupyterlab-workspace
            labworkspacesauto-i-72c8.jupyterlab-workspace
1
2
3
4
5
6
7
8
9
import os 
 
for folderName, subfolders, filenames in os.walk('C:\\Users\\Administrator\\.jupyter'): 
    print('The current folder is ' + folderName) 
    for subfolder in subfolders: 
        print('SUBFOLDER OF ' + folderName + ': ' + subfolder) 
    for filename in filenames: 
        print('FILE INSIDE ' + folderName + ': '+ filename) 
    print('') 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
The current folder is C:\Users\Administrator\.jupyter
SUBFOLDER OF C:\Users\Administrator\.jupyter: lab
FILE INSIDE C:\Users\Administrator\.jupyter: jupyter_notebook_config.py
FILE INSIDE C:\Users\Administrator\.jupyter: migrated

The current folder is C:\Users\Administrator\.jupyter\lab
SUBFOLDER OF C:\Users\Administrator\.jupyter\lab: user-settings
SUBFOLDER OF C:\Users\Administrator\.jupyter\lab: workspaces

--snip--

The current folder is C:\Users\Administrator\.jupyter\lab\workspaces
FILE INSIDE C:\Users\Administrator\.jupyter\lab\workspaces: lab-a511.jupyterlab-workspace
FILE INSIDE C:\Users\Administrator\.jupyter\lab\workspaces: labworkspacesauto-2-f9e3.jupyterlab-workspace
FILE INSIDE C:\Users\Administrator\.jupyter\lab\workspaces: labworkspacesauto-i-72c8.jupyterlab-workspace

Sys

1
2
3
4
5
6
7
import sys
f = [x ** 2 for x in range(1, 100000)]
print(sys.getsizeof(f)) 
# 824464
f = (x ** 2 for x in range(1, 100000))
print(sys.getsizeof(f))
# 120
1
2
3
4
5
6
7
8
import sys 
 
while True: 
    print('Type exit to exit.') 
    response = input() 
    if response == 'exit': 
        sys.exit() 
    print('You typed ' + response + '.')

Extensions

myImages.py

1
2
3
4
5
6
7
picFormats = ['jpg', 'png', 'gif', 'jpeg', 'bmp', 'webp']
print('Plz enter a file name:')
file = input()
if file.lower().split(".")[-1] not in picFormats:
    print('Maybe not an image file. ')
else:
    print(file + ' is an image file.')
1
2
3
4
5
6
7
8
9
def get_suffix(filename, has_dot=False):
    pos = filename.rfind('.')
    if 0 < pos < len(filename) - 1:
        index = pos if has_dot else pos + 1
        return filename[index:]
    else:
        return ''
get_suffix("foo.bar.jpg")
# jpg

Sys.argv

sys.argv[] 可以直接从外部获取参数，举个例子：

py_sys_argv_test.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import sys

def test():
    print(sys.argv)
    args = sys.argv
    if len(args) == 1:
        print('Hello, world!')
    elif len(args) == 2:
        print('Hello, %s!' % args[1])
    else:
        print('Too many arguments!')

if __name__ == '__main__':
    test()
1
2
3
4
5
6
7
8
9
$ python py_sys_argv_test.py
# ['py_sys_argv_test.py']
# Hello, world!
$ python py_sys_argv_test.py Max
# ['py_sys_argv_test.py', 'Max']
# Hello, Max!
$ python py_sys_argv_test.py Max Chloe
# ['py_sys_argv_test.py', 'Max', 'Chloe']
# Too many arguments!

py_sys_argv_lines.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import sys
for arg in sys.argv:
    try:
        # ignore UnicodeDecodeError: 'gbk' codec can't decode byte 0xac in position
        f = open(arg, 'r', errors='ignore')
        s = f.readline()
    except OSError as err:
        print("OS error: {0}".format(err))
    except IOError:
        print('cannot open', arg)
    except:
        print("Unexpected error:", sys.exc_info()[0])
        raise
    else:
        print(arg, 'has', len(f.readlines())+1, 'lines')
        f.close()
1
2
3
4
5
6
7
8
9
10
$ python py_sys_argv_lines.py thoughts.txt
py_sys_lines.py has 15 lines
thoughts.txt has 1 lines

$ python py_sys_argv_lines.py thoughcts.txt
py_sys_lines.py has 15 lines
OS error: [Errno 2] No such file or directory: 'thoughcts.txt'
$ python py_sys_argv_lines.py 红楼梦.txt
py_sys_lines.py has 15 lines
红楼梦.txt has 5541 lines

我们将for arg in sys.argv: 改为 for arg in sys.argv[1:]:，防止统计脚本本身的行数：

1
2
3
$ python py_sys_argv_lines.py 红楼梦.txt Tagore.txt
红楼梦.txt has 5541 lines
Tagore.txt has 8 lines

REFERENCES