Different text standards and encoding methods

While programming and making test you may have perfectly performing programs. But if you add data from various sources pro use libraries like pandas or other which encode or decode text, your programs using text file data may may not perform tasks as programmed. One problem I encountered that pohmeliy.com bot does not read see text data urls which passed through pandas encoding interpreter which in turn enodes everything to UTF-8, which I solved by downloading list using pohmeliy.com tolist bot. And another problem some url lines attaches each to other hindering programmed performance or making list of urls unreadable. Encoding and encoding methods detection is universal problem. It is useful to read for understanding What Every Programmer Absolutely, Positively Needs To Know About Encodings And Character Sets To Work With Text to understand how text works with encodings. To  detect  text encodings I use python chardect
Chardet: The Universal Character Encoding Detector which detects some encodings standards and some variants with detection confidenc. Today I tested I got:

C:\Users\ANTRAS>chardetect C:\Pohmeliy_FB\FB_post_to_groupsCR\lists\CRGroups.txt C:\Pohmeliy_FB\FB_post_to_groupsSILK\lists\SILKGroups.txt
C:\Pohmeliy_FB\FB_post_to_groupsCR\lists\CRGroups.txt: ascii with confidence 1.0
C:\Pohmeliy_FB\FB_post_to_groupsSILK\lists\SILKGroups.txt: ascii with confidence 1.0

So you you have to find one data source or decode and encode  to same byte order variant. About Byte order mark   you can read on Wikipedia.org .

REWRITING, DEBUGGING, TESTING

I have to remind myself the job have done to complete integration of automated program and to test for reliability. The way you organize programming is also important.
Firstly, audit and remind what has been done. In this case I have code of twelve operations;
Secondly, today I found that converting code to colorful html file is practical to visually discern operations and read code. It is great! It will simplify tasks and save time. Similar way I read it in pywinauto shell;
Thirdly, keep in mind to debug errors finding error on command prompt screen and splitting code; Fourthly, to save time I program Autohotkey commands to write repetitive words and paths and open necessary window. It makes you efficient;
Fifth; I will amend this twelve operations codes to simplify working only with downloaded text files, Writing code you add on right code to another. You can watch video of how perform 12 functions.

#python 12345678910OPERACIJA.py
#cd C:\Users\Vartotojas\Desktop\OK1\

import csv
import os
import time
import re
import codecs
import sys
import shutil
import pandas

#PAKEISTI APJUNGTI TEKSTINIUS FAILUS, ISVALYTI BESIDUBLIUOJANCIUS,PASALINTU SU NEREIKALINGAI ZODZIAI

filename1 = 'C:\\Users\\Vartotojas\\Desktop\\OK1\\LINKINPUT.csv'
filename2 = 'C:\\Users\\Vartotojas\\Desktop\\OK1\\LINKINPUT.txt'
filename3 ='C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPSOUTPUT.txt'
filename4 = 'C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPSOUTPUT1.txt'

with open(filename1, 'rb') as inf:
    reader = csv.reader(inf)
    with open(filename2, 'wb') as out:
        writer = csv.writer(out, delimiter='\t')
        writer.writerows(reader)

time.sleep(3)

filename21 = 'LINKINPUT1.txt'


def addUTF8Bom(filename):
  f = codecs.open(filename2, 'r', 'utf-8')
  content = f.read()
  f.close()
  f2 = codecs.open(filename21, 'w', 'utf-8')
  f2.write(u'\ufeff')
  f2.write(content)
  f2.close()

addUTF8Bom(filename2)

shutil.copy(filename21, filename2)
os.remove(filename21)

time.sleep(3)

name = "facebook.com/groups/"

newfile = open(filename3, 'w')
for line in file(filename2, 'r'):
    if name in line:
        newfile.write(line)
newfile.close()

time.sleep(3)

bad_words = ['permalinks']

with open(filename3, 'r') as oldfile, open(filename4, 'w') as newfile:
    for line in oldfile:
        if not any(bad_word in line for bad_word in bad_words):
            newfile.write(line)

time.sleep(3)

#somefile = 'C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPSOUTPUT1.txt'
filename5 = 'C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPGROUPLINK.txt'

with open(filename4, 'r') as infile, open(filename5, 'w') as newfile:
    for line in infile:
        newfile.write(line.split()[0] + '\n')

time.sleep(3)

#somefile = 'C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPSOUTPUT1.txt'
filename6 = 'C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPGROUPNAME.txt'


f = open(filename4, "r")
g = open(filename6, "w")
for line in f:
    if line.strip():
        g.write("\t".join(line.split()[1:]) + "\n")
f.close()
g.close()

time.sleep(3)

filename7 = 'C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPGROUPNAME1.txt'

prefix = ""
suffix = "QQQQQQ"

with open(filename6, 'r') as src:
    with open(filename7, 'w') as dest:
       for line in src:
           dest.write('%s%s%s\n' % (prefix, line.rstrip('\n'), suffix))

time.sleep(3)

filename8 = "C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPGROUPNAME2.txt"

fin = open(filename7, 'r')#PIRMA SUNEMARUS
output = open(filename8, 'w')
text1 = fin.read()
change =  re.sub(r'[^\w]','', text1) #re.sub(r'[^\w\s]','', text1) #REIKALINGA  PIRMAM KARTUI
print change
output.write(change)
fin.close()
output.close()

time.sleep(3)

filename9 = "C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPGROUPNAME3.txt"

fin = open(filename8, 'r')#PIRMA SUNEMARUS
output = open(filename9, 'w')
text1 = fin.read()
change =  re.sub(r'QQQQQQ','\n', text1) #re.sub(r'[^\w\s]','', text1) #REIKALINGA  PIRMAM KARTUI
print change
output.write(change)
fin.close()
output.close()

time.sleep(3)

filename10 = "C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPGROUPLINK1.txt"

fin = open(filename5, 'r')#PIRMA SUNEMARUS
output = open(filename10, 'w')
text1 = fin.read()
change =  re.sub(r'https://www.facebook.com/groups/','', text1) #re.sub(r'[^\w\s]','', text1) #REIKALINGA  PIRMAM KARTUI
print change
output.write(change)
fin.close()
output.close()

time.sleep(3)

filename11 = "C:\\Users\\Vartotojas\\Desktop\\OK1\\FBGROUPGROUPLINK2.txt"

fin = open(filename10, 'r')#PIRMA SUNEMARUS
output = open(filename11, 'w')
text1 = fin.read()
change =  re.sub(r'/','', text1) #re.sub(r'[^\w\s]','', text1) #REIKALINGA  PIRMAM KARTUI
print change
output.write(change)
fin.close()
output.close()

time.sleep(3)

import itertools

filename12 = 'FBIDNAME.txt'

from itertools import izip
with open(filename12, 'w') as res, open(filename11) as f1, open(filename9) as f2:
    for line1, line2 in zip(f1, f2):
        res.write("{} {}\n".format(line1.rstrip(), line2.rstrip()))

time.sleep(3)   

os.remove(filename2)
os.remove(filename3)
os.remove(filename4)
os.remove(filename5)
os.remove(filename6)
os.remove(filename7)
os.remove(filename8)
os.remove(filename9)
os.remove(filename10)
os.remove(filename11)