Search This Blog

Sunday, November 26, 2017

Automated sound engineering software SFS WIN using python pywinauto

The program often bugs on function to save processed filed since this software has commercial version. There is one solution to try is to automate few steps with autohotkey and integrate with pyahk library, or place rande time.sleep method. The software may have internal time meter.

Monday, November 20, 2017

Bots for social media

In my pipeline many projects to facilitate tasks, to automate them to make business intelligence while scrapying data, process with pandas data frames. One high ROI project is completed - automation of computer tasks automation using Facebook suite. You can rent Facebook Suite, find customers and use industrial posting and other tasks platform to market. is python programmed bot, which post randomly in intervals safe for Facebook algorithms. Sounds perfect until you do it. Posting has to generate sales and good visibility to customers, and there should be many for low price to be worth doing it. With platform there is much manual job tasks which makes your life miserable.
Firstly, you have to collect much information of Facebook Groups - do not their purpose, members numbers, policies, if they are administered or not. To be profitable to you have to post to free groups, with high members numbers, to keep policies. To administer I programmed systematization tool and integrate data with pandas for sorting and update. Systematization program tool assigns members numbers to link and rules as well, therefore you can rearrange within one minute scraped from Facebook member profile Groups data to list - group name, members numbers and rules; and afterwards match  with links and names while merging files. The program is great process hundreds of targeted members data and get best plan for marketing with highest visibility at low costs.
Secondly, you have to post preferably up to 20 post and make pauses. So you have to delete first twenty groups. Manual job is required. There are tasks that can be done - collection and updating of moderated, approval, disabled for posting groups. It will help to save postings to improve visibility to potential customers and improve SEO and overall conversions of online business. These tasks have been automated with programs. Thirdly, you should often update your posting messages.
So I looked for  others of cheap traffic and audience source. Search "github python thumblr, twitter, pinterest, " shows many projects which are worth to test.  I will write about them later.

Sunday, November 19, 2017

Different text standards and encoding methods

While programming and making test you may have perfectly performing programs. But if you add data from various sources pro use libraries like pandas or other which encode or decode text, your programs using text file data may may not perform tasks as programmed. One problem I encountered that bot does not read see text data urls which passed through pandas encoding interpreter which in turn enodes everything to UTF-8, which I solved by downloading list using tolist bot. And another problem some url lines attaches each to other hindering programmed performance or making list of urls unreadable. Encoding and encoding methods detection is universal problem. It is useful to read for understanding What Every Programmer Absolutely, Positively Needs To Know About Encodings And Character Sets To Work With Text to understand how text works with encodings. To  detect  text encodings I use python chardect
Chardet: The Universal Character Encoding Detector which detects some encodings standards and some variants with detection confidenc. Today I tested I got:

C:\Users\ANTRAS>chardetect C:\Pohmeliy_FB\FB_post_to_groupsCR\lists\CRGroups.txt C:\Pohmeliy_FB\FB_post_to_groupsSILK\lists\SILKGroups.txt
C:\Pohmeliy_FB\FB_post_to_groupsCR\lists\CRGroups.txt: ascii with confidence 1.0
C:\Pohmeliy_FB\FB_post_to_groupsSILK\lists\SILKGroups.txt: ascii with confidence 1.0

So you you have to find one data source or decode and encode  to same byte order variant. About Byte order mark   you can read on .

Tuesday, November 14, 2017

PYAHK AutoHotKey via Python

AutoHotKey is a powerful task automator with a terrible scripting language built-in. Python is a powerful scripting language with old, unmaintained, and incomplete automation modules. Combining the two creates a powerful automation tool with a powerful scripting back-end with access to all the power of the Python standard library.
This is made possible by the amazing Python ctypes module and the AutoHotKey.dll project. Together they allow exchange of data between both scripting engines, and the execution of AHK commands from Python.
link PYAHK AutoHotKey via Python

Combining with pywinauto and SWAPY  automation tool and  PYAHK AutoHotKey via Python automated various software solutions can be programmed. Before testing I assume that cycle can be programmed using iterate range function, but for sound I have idea to analysis record speech and base iteration number to achieve necessary physics.

Sunday, November 12, 2017

Scrapy Crawling projects Search Github python scrapy

Useful projects for scrapy. I will test or use them for coding my tailor made. These crawlers do not include some pypi projects llike imagebot. 

scrapy-tdd 0.1.3 : Python Package Index - PyPI Helpers and examples to build Scrapy Crawlers in a test driven way. 
The project to be tested firstly.
For pypi scrapy crawlers I used search "pypi scrapy crawlers".

Today I code Pandas data import and updating and sorting by numbers as well strings and dropping repetitive lines. The day was successful. To be more productive I will program scrapy codes to collect information for freelancing.

Scrapy, a fast high-level web crawling & scraping framework for Python.
GitHub - geekan/scrapy-examples: Multifarious Scrapy examples .
scrapy-examples - Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.

GitHub - scrapy/dirbot: Scrapy project to scrape public web directories.
dirbot - Scrapy project to scrape public web directories (educational) [DEPRECATED]

GitHub - scrapy/quotesbot: Quotebot for one bot.
This is a Scrapy project to scrape quotes from famous people from (github repo). This project is only meant for educational purposes.

GitHub - rmax/scrapy-redis: Redis-based components for Scrapy.
Redis-based components for Scrapy. Free software: MIT license; Documentation:

GitHub - scrapy/scrapely: A pure-python HTML screen-scraping library
A pure-python HTML screen-scraping library. Contribute to scrapely development by creating an account on GitHub.

GitHub - mjhea0/Scrapy-Samples: Scrapy examples crawling Craigslist
Scrapy examples crawling Craigslist. Contribute to Scrapy-Samples development by creating an account on GitHub.

GitHub - scrapinghub/portia: Visual scraping for Scrapy
Visual scraping for Scrapy. Contribute to portia development by creating an account on GitHub.

GitHub - edx/pa11ycrawler: Python crawler (using Scrapy) that uses.

pa11ycrawler - Python crawler (using Scrapy) that uses Pa11y to check accessibility of pages as it crawls.
GitHub - eloyz/reddit: .
2015-02-05 - Scrapy (Python Framework) Example using

GitHub - vinta/BlackWidow: Web crawler using Scrapy

Web crawler using Scrapy Install. $ sudo apt-get install python-dev libxml2-dev libxslt1-dev $ pip install -r requirements.txt.

GitHub - istresearch/scrapy-cluster: This Scrapy project uses Redis.
scrapy-cluster - This Scrapy project uses Redis and Kafka.

GitHub - scrapy/w3lib: Python library of web-related functions
Python library of web-related functions. Contribute to w3lib development by creating an account on GitHub.

GitHub - scrapy-plugins/scrapy-deltafetch: Scrapy spider middleware.

scrapy-deltafetch - Scrapy spider middleware to ignore requests to pages containing items ... DeltaFetch middleware depends on Python's bsddb3 package.

GitHub - scrapy/scrapyd: A service daemon to run Scrapy spiders
A service daemon to run Scrapy spiders. Scrapyd is a service for running Scrapy spiders.

Scrapy Plugins · GitHub
Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls.

Web Scraping with Scrapy and MongoDB - Real Python

Deploy your Scrapy Spiders from GitHub – The Scrapinghub Blog
2017-04-19 - Up until now, your deployment process using Scrapy Cloud has probably ...
Scrapy Cloud's new GitHub integration will help you ensure that your.

python - Scrapy and github login - Stack Overflow
2016-11-26 - You shall try like this def parse(self, response): print "in parse function" yield FormRequest.from_response( response, ...

Running scrapy spider programmatically - Musings of a programmer
Please check the project on github. The Scrapy Spider : It is a python class in the scrapy framework that is responsible for fetching URLs and parsing the

scrapy-crawlera 1.2.4 : Python Package Index
Crawlera middleware for Scrapy. scrapy-crawlera 1.2.4 .Author: Raul Gallegos; Home Page:;

Webscraping Airbnb with scrapy – - Latest Posts
You can find the complete code here as github repo, feel free to fork, clone.

Scrapy Tutorial: Web Scraping Craigslist – Web Scraping with Python
Craigslist Scrapy Tutorial on GitHub - You can also find all the spiders we explained in this Python Scrapy tutorial on GitHub.

How do I change the data type of a pandas Series?

Friday, November 10, 2017

Pandas Data frame update update

Pandas data frame update should be one of the most useful functions.

Modify DataFrame in place using non-NA values from passed DataFrame. Aligns on indices

Pandas dataframe merge with update data

Pandas dataframe merge with update data
print (pd.concat([df1, df2]).drop_duplicates('date', keep='last'))
date value
0 2017-01-01 1
1 2017-01-02 1
2 2017-01-03 1
0 2017-01-04 2
1 2017-01-05 2
2 2017-01-06 2
3 2017-01-07 2
4 2017-01-08 2
You can use concat with drop_duplicates by column date and keep last values:
Concat works, but drop_dublicates('ID', keep='last') does not.

Wednesday, November 8, 2017

Python scripts for search engines.
Search engine scripts, some are really useful. To be tested. And Copernicus for download information from social media. OSINT - Open Source Intelligence "Leading Intelligence and Investigation Technique"
New! Automatically Discover Website Connections Through Tracking Codes

Python examples Timeline

python_ftp.pyclearinterfacepythonpython ftp code2017-11-08 - p=1262017-11-08
http_request.pynhrwtrhttp requestHTTPの勉強2017-11-08
autowihtelaamalifautowihteauto white correction opencv2017-11-08
Queue.pybosskunqueueQueue2017-11-08 Exercise 1 + bonus objectives2017-11-08
ghost_in_the_heap.pypeternguyen93ghost heap2017-11-08
dummy-web-server.pysherlockwudummy servera minimal http server in python. Responds to GET, HEAD, POST requests, but will fail on anything else.2017-11-08 Exercise 2 + bonus objectives2017-11-08
gcloud_bq_tables.pykrnrgcloud tablesexample of BigQuery tables creation2017-11-08
ytdl.pywDimChtzytdlPython script (GUI) to download YouTube videos (as video or audio).2017-11-08
guillet.pydcw32guilletGuillet Data2017-11-08
focus_group.pySumeniafocus groupfocus group2017-11-08
genie.pyminhtt159genieSvATTT2017 - Crypto22017-11-08
jsreport.pyrylincolnjsreportpython jsreport batch2017-11-08
pair_checker.pychewingFpair checker2017-11-08

Sunday, November 5, 2017

Useful sleepless night because of neighbors night became useful since I systemized Python source ideas

Because of noise and moved furniture in neighbor flat and felt stress and though about projects if I have sleepless nights. Psychologist assert that minor short-term stress are useful for innovative or creative ideas. I was concerned about organization capabilities. Python has more than 100 thousand projects and libraries published, "do  not invent bicycles" is applicable.

Main ideas to write new code:
1. Systemize your ideas codes, i.e. my tree of codes for functions. For examples - copied or developed codes for data processing, ext file automation and etc. Base system on ready to integrate solution codes and vice versa on specific narrow (for example string operation) with necessary description how adapt and develop code. For this purpose draft tree for python codes folders structures.
2. Secondly I download python books (O'Reily free python books, Python Cookbook, Python Essential Reference and etc.. Systemize book and write description for solutions. Will it be faster to find with relevant searches on stackoverflow, online tutorials, code search engines? I do not know. I just know that you will grasp ideas of may be several or numerous application possibilities, or may be alternative and write code search wording using technical term.
3. Exploit with priority pypi and github, to find projects or copy necessary part of codes. Systemize sources of urls with bookmarks. Programming is dynamic, the code within one year can become obselete.
4. Systemize specific libraries sources and tutorials for specific needs and exploit.

The system is needed to automate tasks, jobs; so I look for ways to streamline/automate code creations. Also I can reduce time needed to learn programming and find solutions.

So on github I found projects for codes to save time.
  • Project 1 - E-mail and phone number extractor
  • Project 3 - Regular expression version for strip()
  • Project 8 - Backup a file/folder into a zip file
  • project 13 - Copying '.jpg' and '.pdf' files into a newfolder

6 Useful Python Libraries Recommended by

New libraries for automation:

RoboBrowser: Your friendly neighborhood web scraper

Requests: HTTP for Humans Library to read and test/

Collection / Learn Python  List of sources to read, test.
Collection /

Unexpectedly I found searching solutions Python documentation file with cheatsheets. This site contains cheatsheets, commands summaries and other helpful information to rehearse. I stumbled to problem how to read text file with string.lsplit method (and rstrtip and so on). This kind of text files manipulation is necessary to prepare data for Pandas frame since because of text markups I get undesirable underscore beginning first line character. So I though about systemizing file read and after processing write methods. I use at least three read writing to methods and rewriting. Also parsing (reading) or looping through lines methods. And why they differ, when to use them since there are many methods with different parameters requirements and modules (libraries) with own environments (codings). This help to streamline automation of data processing and writing out results. For example regex library may require one method, python commands another. Most popular libraries preferbaly be systemized. look for:

Processing Text in Python
Parsing text files in Python
Reading text files in Python
Looping text files in Python

Thursday, November 2, 2017

Pytrends for small business by General Mills Minneapolis

On github I found tools  pytrends for small business to use Google trends developed by General Mills Minneapolis, MN (Big Name).

Unofficial API for Google Trends

Allows simple interface for automating downloading of reports from Google Trends. Main feature is to allow the script to login to Google on your behalf to enable a higher rate limit. Only good until Google changes their backend again :-P. When that happens feel free to contribute!

The following API methods are available:

Interest Over Time: returns historical, indexed data for when the keyword was searched most as shown on Google Trends' Interest Over Time section.

Interest by Region: returns data for where the keyword is most searched as shown on Google Trends' Interest by Region section.

Related Topics: returns data for the related keywords to a provided keyword shown on Google Trends' Related Topics section.

Related Queries: returns data for the related keywords to a provided keyword shown on Google Trends' Related Queries section.

Trending Searches: returns data for latest trending searches shown on Google Trends' Trending Searches section.

Top Charts: returns the data for a given topic shown in Google Trends' Top Charts section.

Suggestions: returns a list of additional suggested keywords that can be used to refine a trend search.

With red is report dates that I cannot change, I gues it is 5 years period start.

C:\Users\ANTRAS\Desktop> python
            pizza  bagel isPartial
2012-11-04     56      1     False
2012-11-11     55      1     False
2012-11-18     54      1     False
2012-11-25     56      1     False
2012-12-02     57      1     False
           pizza  bagel
Argentina     19      0
Australia     69      1
Austria       25      0
Belgium       34      0
Brazil        24      0
       date                                         exploreUrl  \
0  20171101      /trends/explore#q=Academy&date=now+7-d&geo=US
1  20171101  /trends/explore#q=brian+hoyer&date=now+7-d&geo=US
2  20171101  /trends/explore#q=d%C3%ADa+de+muertos&date=now...
3  20171101  /trends/explore#q=Donna+Brazile&date=now+7-d&g...
4  20171101  /trends/explore#q=spectrum+internet&date=now+7...

      formattedTraffic hotnessColor  hotnessLevel  \
0  Daugiau nei 100 000      #e68033           3.0
1   Daugiau nei 50 000      #f0a049           2.0
2   Daugiau nei 50 000      #f0a049           2.0
3   Daugiau nei 50 000      #f0a049           2.0
4   Daugiau nei 50 000      #f0a049           2.0

                                          imgLinkUrl  \

                      imgSource  \
0  Atlanta Journal Constitution
1                   ESPN (blog)
2                 Mission Local
3                      HuffPost
4     Austin American-Statesman

                                              imgUrl  \
0  //
1  //
2  //
3  //
4  //

                                    newsArticlesList relatedSearchesList  \
0  [{u'snippet': u'An officer was hit early Thurs...                  []
1  [{u'snippet': u'As part of the 2019 salary, <b...                  []
2  [{u'snippet': u'All month, residents and touri...                  []
3  [{u'snippet': u'<b>Donna Brazile</b>, former i...                  []
4  [{u'snippet': u'Earlier: <b>Spectrum</b> custo...                  []

   safe                                           shareUrl     startTime  \
0   1.0  1.509628e+09
1   1.0  1.509624e+09
2   1.0  1.509631e+09
3   1.0  1.509638e+09
4   1.0  1.509624e+09

               title                                   titleLinkUrl  \
0            Academy              //
1        Brian Hoyer          //
2     Día de Muertos  //
3      Donna Brazile        //
4  Spectrum internet    //

0                 100000.0
1                  50000.0
2                  50000.0
3                  50000.0
4                  50000.0
   delta                       deltaSummary  \
0    3.0  <b> + 3</b> nuo pra?jusio m?nesio
1    1.0  <b> + 1</b> nuo pra?jusio m?nesio
2    2.0  <b> + 2</b> nuo pra?jusio m?nesio
3    NaN            <b>Nauja</b> diagramoje
4   -3.0    <b>-3</b> nuo pra?jusio m?nesio

                                         description  \
0  {u'source': u'Wikipedia', u'sourceUrl': u'http...
1  {u'source': u'Wikipedia', u'sourceUrl': u'http...
2  {u'source': u'Wikipedia', u'sourceUrl': u'http...
3  {u'source': u'Wikipedia', u'sourceUrl': u'http...
4  {u'source': u'Wikipedia', u'sourceUrl': u'http...

                                          exploreUrl hotnessColor  \
0  /trends/explore?q=/m/09gkdy4&date=2016-11-1+20...      #d04108
1  /trends/explore?q=/m/0214xp&date=2016-11-1+201...      #d04108
2  /trends/explore?q=/m/0gs6vr&date=2016-11-1+201...      #d04108
3  /trends/explore?q=/m/01xwv7&date=2016-11-1+201...      #d04108
4  /trends/explore?q=/m/0bdxs5&date=2016-11-1+201...      #d04108

   hotnessLevel   idForTracking  isNew                    peakRankSummary  \
0             5   Ariana Grande  False  Pakilo iki <b>#1</b> (per m?nes?)
1             5     Sunny Leone  False  Pakilo iki <b>#2</b> (per m?nes?)
2             5    Selena Gomez  False  Pakilo iki <b>#1</b> (per m?nes?)
3             5  Dave Chappelle   True  Pakilo iki <b>#3</b> (per m?nes?)
4             5     Miley Cyrus  False  Pakilo iki <b>#1</b> (per m?nes?)

                                            shareUrl  \

                          timePeriodsInChartSummary           title  \
0      <b>51 m?nuo</b> 10 populiariausi? diagramoje   Ariana Grande
1    <b>17 m?nesi?</b> 10 populiariausi? diagramoje     Sunny Leone
2  <b>105 m?nesiai</b> 10 populiariausi? diagramoje    Selena Gomez
3    <b>9 m?nesiai</b> 10 populiariausi? diagramoje  Dave Chappelle
4     <b>121 m?nuo</b> 10 populiariausi? diagramoje     Miley Cyrus

   titleLength                              titleLinkUrl trafficSummary  \
0           13   //            100
1           11     //             84
2           12    //             81
3           14  //             81
4           11     //             80

0  Ariana Grande yra Nr. 1 ?Google? tendencij? di...
1  Sunny Leone yra Nr. 2 ?Google? tendencij? diag...
2  Selena Gomez yra Nr. 3 ?Google? tendencij? dia...
3  Dave Chappelle yra Nr. 4 ?Google? tendencij? d...
4  Miley Cyrus yra Nr. 5 ?Google? tendencij? diag...
[{u'type': u'Dish', u'mid': u'/m/0663v', u'title': u'Pizza'}, {u'type': u'Restaurant company', u'mid': u'/m/09cfq', u'title': u'Pizza Hut'}, {u'type': u'Topic', u'mid': u'/m/05m1lt', u'title': u'Pizza delivery'}, {u'type': u'Restaurant company', u'mid': u'/m/03clwm', u'title': u"Domino's Pizza"}, {u'type': u'Fast food restaurant company', u'mid': u'/m/021sqj', u'title': u'Pizza Pizza'}]

So I tested my keywords to find best images SEO possibilities in England Norther town. green lines show best search traffic concentrate (competitiveness aside). This tool increase productivity after all, since you can also search Youtube, shoppers site also.

C:\Users\ANTRAS\Desktop> python
            flowers delivery Manchester  flowers Manchester  \
2012-11-04                            0                   0
2012-11-11                            0                 100
2012-11-18                            0                   0
2012-11-25                            0                   0
2012-12-02                            0                   0

            send flowers Manchester  flowers of Manchester  \
2012-11-04                        0                      0
2012-11-11                        0                      0
2012-11-18                        0                      0
2012-11-25                        0                      0
2012-12-02                        0                      0

            flowers in Manchester isPartial
2012-11-04                      0     False
2012-11-11                      0     False
2012-11-18                      0     False
2012-11-25                      0     False
2012-12-02                      0     False

         flowers delivery Manchester  flowers Manchester  \
England                           15                 100

         send flowers Manchester  flowers of Manchester  flowers in Manchester
England                        0                      8                     31