How to traverse through the files in a directory?

Posted on

Question :

How to traverse through the files in a directory?

I have a directory logfiles. I want to process each file inside this directory using a Python script.

for file in directory:
      # do something

How do I do this?

Asked By: Bruce


Answer #1:

With os.listdir() or os.walk(), depending on whether you want to do it recursively.

Answer #2:

In Python 2, you can try something like:

import os.path

def print_it(x, dir_name, files):
    print dir_name
    print files

os.path.walk(your_dir, print_it, 0)

Note: the 3rd argument of os.path.walk is whatever you want. You’ll get it as the 1st arg of the callback.

In Python 3 os.path.walk has been removed; use os.walk instead. Instead of taking a callback, you just pass it a directory and it yields (dirpath, dirnames, filenames) triples. So a rough equivalent of the above becomes

import os

for dirpath, dirnames, filenames in os.walk(your_dir):
    print dirpath
    print dirnames
    print filenames
Answered By: luc

Answer #3:

You can list every file from a directory recursively like this.

from os import listdir
from os.path import isfile, join, isdir

def getAllFilesRecursive(root):
    files = [ join(root,f) for f in listdir(root) if isfile(join(root,f))]
    dirs = [ d for d in listdir(root) if isdir(join(root,d))]
    for d in dirs:
        files_in_d = getAllFilesRecursive(join(root,d))
        if files_in_d:
            for f in files_in_d:
    return files
Answered By: Matheus Araujo

Answer #4:

import os
# location of directory you want to scan
loc = '/home/sahil/Documents'
# global dictonary element used to store all results
global k1 
k1 = {}

# scan function recursively scans through all the diretories in loc and return a dictonary
def scan(element,loc):

    le = len(element)

    for i in range(le):   

            second_list = os.listdir(loc+'/'+element[i])
            temp = loc+'/'+element[i]
            print "....."
            print "Directory %s " %(temp)
            print " "
            print second_list
            k1[temp] = second_list

        except OSError:

    return k1 # return the dictonary element    

# initial steps
    initial_list = os.listdir(loc)
    print initial_list
except OSError:
    print "error"

k =scan(initial_list,loc)
print " ..................................................................................."
print k

I made this code as a directory scanner to make a playlist feature for my audio player and it will recursively scan all the sub directories present in directory.

Answered By: pyth

Answer #5:

You could try glob:

import glob

for file in glob.glob('log-*-*.txt'):
  # Etc.

But glob doesn’t work recursively (as far as I know), so if your logs are in folders inside of that directory, you’d be better off looking at what Ignacio Vazquez-Abrams posted.

Answered By: Blender

Answer #6:

If you need to check for multiple file types, use

glob.glob("*.jpg") + glob.glob("*.png")

Glob doesn’t care about the ordering of the files in the list. If you need files sorted by filename, use

Answered By: ATOzTOA

Answer #7:

import os
rootDir = '.'
for dirName, subdirList, fileList in os.walk(rootDir):
    print('Found directory: %s' % dirName)
    for fname in fileList:
        print('t%s' % fname)
    # Remove the first entry in the list of sub-directories
    # if there are any sub-directories present
    if len(subdirList) > 0:
        del subdirList[0]

Answered By: aqua

Answer #8:

Here’s my version of the recursive file walker based on the answer of Matheus Araujo, that can take optional exclusion list arguments, which happens to be very helpful when dealing with tree copies where some directores / files / file extensions aren’t wanted.

import os

def get_files_recursive(root, d_exclude_list=[], f_exclude_list=[], ext_exclude_list=[], primary_root=None):
Walk a path to recursively find files
Modified version of that includes exclusion lists
:param root: path to explore
:param d_exclude_list: list of root relative directories paths to exclude
:param f_exclude_list: list of filenames without paths to exclude
:param ext_exclude_list: list of file extensions to exclude, ex: ['.log', '.bak']
:param primary_root: Only used for internal recursive exclusion lookup, don't pass an argument here
:return: list of files found in path

# Make sure we use a valid os separator for exclusion lists, this is done recursively :(
d_exclude_list = [os.path.normpath(d) for d in d_exclude_list]

files = [os.path.join(root, f) for f in os.listdir(root) if os.path.isfile(os.path.join(root, f))
         and f not in f_exclude_list and os.path.splitext(f)[1] not in ext_exclude_list]
dirs = [d for d in os.listdir(root) if os.path.isdir(os.path.join(root, d))]
for d in dirs:
    p_root = os.path.join(primary_root, d) if primary_root is not None else d
    if p_root not in d_exclude_list:
        files_in_d = get_files_recursive(os.path.join(root, d), d_exclude_list, f_exclude_list, ext_exclude_list, primary_root=p_root)
        if files_in_d:
            for f in files_in_d:
                files.append(os.path.join(root, f))
return files
Answered By: Orsiris de Jong

Leave a Reply

Your email address will not be published. Required fields are marked *