need to pass python vars to bash avconv command

Question

bash hacker here getting my feet wet with python. I have almost ported one of my bash scripts to python, but I cannot seem to figure out how to pass python vars to the bash command avconv to convert .ogg audio files to .mp3 files (sigh mp3 player does not play .ogg).

anyway I thought I would paste what I have sofar, and it is all working as desired until the last four lines of 'if' in final 'for' loop. Of course bash/avconv doesn't recognize python var 'f' passed from python 'for' loop, and not sure how to go about doing so.

#!/usr/bin/python

#genre = GET SONG GENRE DIR
#if genre == rock+
#    sGenreDir = /media/multiMediaA_intHdA720Gb/music/heavierAlt

import os

def get_filepaths(directory):

    #This function will generate the file names in a directory tree by walking the tree either top-down or bottom-up. For each directory in the tree rooted at directory top (including top itself), it yields a 3 tuple (dirpath, dirnames, filenames).

    file_paths = []     # create list to store full paths to each song contained recursively within

    # walk the tree
    for root, directories, files in os.walk(directory):

        for filename in files:

            # join the two strings in order to know the fill file path
            filepath = os.path.join(root, filename)

            file_paths.append(filepath)     # add it to the list

    return file_paths

# run the avove function and store its results in a variable
#full_file_paths = get_filepaths("/media/multiMediaA_intHdA720Gb/music/heavierAlt")
full_file_paths = get_filepaths("/media/multiMediaA_intHdA720Gb/music/rockAndHeavier")

#print len(full_file_paths)

cntr = 0
for f in full_file_paths:

    if not (f.endswith(".mp3") or f.endswith(".ogg")):
        del full_file_paths[cntr]

    cntr = cntr + 1

#print len(full_file_paths)

# create random song list avoiding duplicate songs
destDir =  "/home/keithpc/tmp/rsList"    # will be selected by user
numSongs = 10   # number of songs will be chosen by the user
songList = []   # list to hold randomly selected songs
cntr = 1    # that will be incremented for each song appended to list 'songList'
while cntr <= numSongs:    #begin random song selection/list creation
    from random import choice
    newSong = choice(full_file_paths)

    if not newSong in songList:    # 'if' newSong is not yet list element then add it and ++cntr
        songList.append(newSong)
        cntr = cntr + 1

print len(songList)
for f in songList:
    print f

for f in songList:
    if f.endswith(".mp3"):    # cp .mp3 to player
        import shutil
        shutil.copy2(f, destDir)
    else:    # need to avconv .ogg before copying to player
        import os
        cmd = 'avconv -i f -c:a libmp3lame -q:a 4 -map_metadata 0:s $destDir/f/%ogg/mp3'
        os.system(cmd)

UPDATED: pasting only the last section of above script dealing with random song selection that attempts to avconv 'ogg' to '.mp3' on the fly copying from src to dest

# begin to compile random song list
#destDir =  "/home/keithpc/tmp/rsList"   # will be var supplied by user ; employed in 'for' loop below instead
numSongs = 3   # number of songs var will be supplied by user
songList = []   # list to hold randomly selected songs
cntr = 1    # that will be incremented for each song appended to list 'songList'
while cntr <= numSongs:     # 'while' loop to iterate until 'numSongs' is matched by 'cntr'
    from random import choice
    newSong = choice(lFullFilePaths)    # randomly selected song from list 'lFullFilePaths' to add to list 'songList' if it is not a duplicate

    if not newSong in songList:     # 'if' var does not exist in list 'songList' then add it and cntr++
        songList.append(newSong)
        cntr = cntr + 1

#print len(songList)
#for f in songList:
#    print f

for f in songList:
    destDir =  "/home/keithpc/tmp/rsList"   # will be var supplied by user ; needs to reset each iteration or else it concantanates with previous song details
    if f.endswith(".mp3")   # 'if' song type is '.mp3' then copy direct to mp3 player
        import shutil
        shutil.copy2(f, destDir)
    else:   # 'else' song type must be '.ogg' and so must be avcon'd to mp3 on the fly to mp3 player
        import re   # regex function to extract the song name from its path
        sName = re.sub(r'^.*/', r'', f)     # extract song name from it's path
        #print sName
        import subprocess
        destDir = destDir + "/" + sName + "/%ogg/mp3"   # create single var 'destDir' containing args bash/avconv should need to convert '.ogg' sending its output as '.mp3' to mp3 player
        print(destDir)
        subprocess.call(['avconv', '-i', '%s' % f, '-c:a', 'libmp3lame', '-q:a', '4', '-map_metadata', '0:s', '%s' % destDir])

Terminal output when the final 'subprocess' command is commented out reads like:

07:54 python $ ./ranS*.py
3
/media/multiMediaA_intHdA720Gb/music/rockAndHeavier/Radiohead/The_Best_Of/02_Paranoid_Android.ogg
/media/multiMediaA_intHdA720Gb/music/rockAndHeavier/Rod_Stewart/If_We_Fall_In_Love_Tonight/
15_All_For_Love__With_Bryan_Adams_And_Sting.ogg
/media/multiMediaA_intHdA720Gb/music/rockAndHeavier/Smashing_PumpkinsThe/Siamese_Dream/12_sweet_sweet.ogg
02_Paranoid_Android.ogg
/home/keithpc/tmp/rsList/02_Paranoid_Android.ogg/%ogg/mp3
15_All_For_Love__With_Bryan_Adams_And_Sting.ogg
/home/keithpc/tmp/rsList/15_All_For_Love__With_Bryan_Adams_And_Sting.ogg/%ogg/mp3
12_sweet_sweet.ogg
/home/keithpc/tmp/rsList/12_sweet_sweet.ogg/%ogg/mp3

Which looks about right(ish).

Terminal output with final 'subprocess' uncommented reads like:

07:57 python $ ./ranS*.py
3
/media/multiMediaA_intHdA720Gb/music/rockAndHeavier/Guns_N_Roses/Patience_Live_At_The_Ritz_EP_Japanese/03_I_Used_To_Love_Her.ogg
/media/multiMediaA_intHdA720Gb/music/rockAndHeavier/SmithsThe/The_Best_Of_The_Smiths_Vol_1/05_girlfriend_in_a_coma_the_best_of_the_smiths_vol._1.mp3
/media/multiMediaA_intHdA720Gb/music/rockAndHeavier/Oasis/The_Masterplan/14_The_Masterplan.ogg
03_I_Used_To_Love_Her.ogg
/home/keithpc/tmp/rsList/03_I_Used_To_Love_Her.ogg/%ogg/mp3
avconv version 0.8.9-6:0.8.9-1, Copyright (c) 2000-2013 the Libav developers
  built on Nov  3 2013 02:10:51 with gcc 4.7.2
Input #0, ogg, from '/media/multiMediaA_intHdA720Gb/music/rockAndHeavier/Guns_N_Roses/Patience_Live_At_The_Ritz_EP_Japanese/03_I_Used_To_Love_Her.ogg':
  Duration: 00:02:48.69, start: 0.000000, bitrate: 186 kb/s
    Stream #0.0: Audio: vorbis, 44100 Hz, stereo, s16, 192 kb/s
    Metadata:
      ARTIST          : Guns N' Roses
      ALBUM           : Patience (Live At The Ritz) (EP - Japanese)
      TITLE           : I Used To Love Her
      DATE            : 1989
      GENRE           : Rock
      track           : 03
      CDDB            : 680adf09
Unable to find a suitable output format for '/home/keithpc/tmp/rsList/03_I_Used_To_Love_Her.ogg/%ogg/mp3'
14_The_Masterplan.ogg
/home/keithpc/tmp/rsList/14_The_Masterplan.ogg/%ogg/mp3
avconv version 0.8.9-6:0.8.9-1, Copyright (c) 2000-2013 the Libav developers
  built on Nov  3 2013 02:10:51 with gcc 4.7.2
Input #0, ogg, from '/media/multiMediaA_intHdA720Gb/music/rockAndHeavier/Oasis/The_Masterplan/14_The_Masterplan.ogg':
  Duration: 00:05:22.80, start: 0.000000, bitrate: 197 kb/s
    Stream #0.0: Audio: vorbis, 44100 Hz, stereo, s16, 192 kb/s
    Metadata:
      ARTIST          : Oasis
      ALBUM           : The Masterplan
      TITLE           : The Masterplan
      DATE            : 1998
      GENRE           : Rock
      track           : 14
      CDDB            : c00fd90e
Unable to find a suitable output format for '/home/keithpc/tmp/rsList/14_The_Masterplan.ogg/%ogg/mp3'

Perhaps the issue is with the /%ogg/mp3 that attempts to replace '.ogg' ext with '.mp3' ext as part of avconv process? Gawd it looks close, if anyone might have any suggs to help get me homefree.

BTW I have been scouring the inet, and stackoverflow in particular, for applicable help. So I have RTFM before askign for help. I just can't seem to get this last part interacting with bash shell to go.

thanks, nap

Answer 1

A while back I created a similar script to convert a ton of videos into a format that could be played on one of my gaming consoles. Here is what I would change in your current code:

As Martijn Pieters recommended I wouldn't use os.system to call avconv, instead try this:

for f in songList:
    if f.endswith(".mp3"):    # cp .mp3 to player
        import shutil
        shutil.copy2(f, destDir)
    else:    # need to avconv .ogg before copying to player
        import subprocess
        subprocess.call(['avconv', '-i', '%s' % f, '-c:a',
                         'libmp3lame', '-q:a', '4', '-map_metadata',
                         '0:s', '$destDir/f/%ogg/mp3'])

Here is a brief description of changes.. First instead of os.system the code is now using subprocess.call which was imported instead of os. Second I reformatted the command being passed to avconv to use '%s' % f instead of straight f, this will tell Python "Hey, use what is stored in f to fill in this blank!", whereas the original code was saying "Python, use f here!". While reformatting the command being passed you will notice that each segment is contained in quotes, this is due to how subprocess interprets the arguments passed to it. I am sure someone else can explain the whys and hows MUCH better than I can, but simply put there will be a space automatically added in between each argument.

--EDIT--

Ok, after reviewing the updated information I went back through and tweaked the code provided. I was not able to fully test this as I am not at home and currently do not have avconv or the ability to install it on this computer. I did however substitute the shutil and subprocess.call lines with print statements and everything "appears" to work.

Here is the code, a brief explanation of the changes will be below.

#!/usr/bin/python

import os
import subprocess
import shutil
from random import choice

def get_filepaths(directory):
    filenames = {}
    for root, directories, files in os.walk(directory):
        for file in files:
            filenames[file] = os.path.join(root, file)
    return filenames

def remove_file(d, key):
    r = dict(d)
    del r[key]
    return r

full_file_paths = get_filepaths("/media/multiMediaA_intHdA720Gb/music/rockAndHeavier")

to_remove = []

for file, file_path in full_file_paths.items():
    if not (file.endswith('.mp3') or file.endswith('.ogg')):
        to_remove.append(file)

for item in to_remove:
    full_file_paths = remove_file(full_file_paths, item)

destDir = "/home/keithpc/tmp/rsList"
songList = []
numSongs = 10
cntr = 1

while cntr <= numSongs:
    newSong = choice(full_file_paths.keys())
    if newSong not in songList:
        songList.append(newSong)
        cntr += 1

for f in songList:
    finalDest = destDir + '/%ogg/mp3'
    if f.endswith('.mp3'):
        shutil.copy2(full_file_paths.get(f), '%s/%s' %(destDir, f))
    else:
        subprocess.call(['avconv', '-i', '%s' % full_file_paths.get(f), '-c:a',
            'libmp3lame', '-q:a', '4', '-map_metadata', '0:s',
            '%s/%s.mp3' % (finalDest, f.strip('ogg'))])

The first change made was in the get_filepaths definition, and I feel makes renaming a little easier in the end. Instead of building a list as before it creates a dictionary using the files actual filename as a key with the value being the entire path to the file.

Next was

to_remove = []

def remove_file(d, key):
    r = dict(d)
    del r[key]
    return r

for file, file_path in full_file_paths.items():
    if not (file.endswith('.mp3') or file.endswith('.ogg')):
        to_remove.append(file)

for item in to_remove:
    full_file_paths = remove_file(full_file_paths, item)

This is similar to how you were original removing unwanted files but rewritten to handle a dictionary instead.

The last change was to the final for loop. This adjustment is primarily due to me forgetting an important thing with avconv (formally known as FFmpeg), and that is avconv requires us to tell it what the destination filename will be. If I recall correctly shutil also needs to know the destination filename, at least it didn't work correctly on Windows.

for f in songList:
    finalDest = destDir + '/%ogg/mp3'
    if f.endswith('.mp3'):
        shutil.copy2(full_file_paths.get(f), '%s/%s' %(destDir, f))
    else:
        subprocess.call(['avconv', '-i', '%s' % full_file_paths.get(f), '-c:a',
            'libmp3lame', '-q:a', '4', '-map_metadata', '0:s',
            '%s/%s.mp3' % (finalDest, f.strip('.ogg'))])

Hope this helps.

Answer 2

Ok here it is, I figured out the prob. I rather than using the bash switch /%ogg/mp3 to change file extension during avconv process I need rather to create a 'sNameOgg' var and then use python regex-sub (I am a big-time SED'r so I am glad I discovered the python way to do it instead, but still a little crytic(ish) atm) to create var 'sNameMp3' and passing that along in modified var 'destDir' included in final subprocess.call.

else:
    import re   # regex function to extract the song name from its path
    sNameOgg = re.sub(r'^.*/', r'', f)
    #print sNameOgg
    sNameMp3 = re.sub(r'\.ogg', r'.mp3', sNameOgg)
    #print sNameMp3
    import subprocess
    destDir = destDir + "/" + sNameMp3    # + "/%ogg/mp3"
    print destDir
    subprocess.call(['avconv', '-i', '%s' % f, '-c:a', 'libmp3lame', '-q:a', '4', '-map_metadata', '0:s', '%s' % destDir])

Things are still a little hazy for me atm with regards to the entire script as I basically put it together one piece/chunk/block at a time, and I still need to incorporate user input functions, but looking at the working result it reads a heck of a lot cleaner than its bash counterpart and that makes me happy. Sweet!