The program often bugs on function to save processed filed since this software has commercial version. There is one solution to try is to automate few steps with autohotkey and integrate with pyahk library, or place rande time.sleep method. The software may have internal time meter.
Sunday, November 26, 2017
Monday, November 20, 2017
Bots for social media
In my pipeline many projects to facilitate tasks, to automate them to make business intelligence while scrapying data, process with pandas data frames. One high ROI project is completed - automation of computer tasks automation using pohmeliy.com Facebook suite. You can rent pohmeliy.com Facebook Suite, find customers and use industrial posting and other tasks platform to market. Pohmeliy.com is python programmed bot, which post randomly in intervals safe for Facebook algorithms. Sounds perfect until you do it. Posting has to generate sales and good visibility to customers, and there should be many for low price to be worth doing it. With pohmeliy.com platform there is much manual job tasks which makes your life miserable.
Firstly, you have to collect much information of Facebook Groups - do not their purpose, members numbers, policies, if they are administered or not. To be profitable to you have to post to free groups, with high members numbers, to keep policies. To administer I programmed systematization tool and integrate data with pandas for sorting and update. Systematization program tool assigns members numbers to link and rules as well, therefore you can rearrange within one minute scraped from Facebook member profile Groups data to list - group name, members numbers and rules; and afterwards match with links and names while merging files. The program is great process hundreds of targeted members data and get best plan for marketing with highest visibility at low costs.
Secondly, you have to post preferably up to 20 post and make pauses. So you have to delete first twenty groups. Manual job is required. There are tasks that can be done - collection and updating of moderated, approval, disabled for posting groups. It will help to save postings to improve visibility to potential customers and improve SEO and overall conversions of online business. These tasks have been automated with programs. Thirdly, you should often update your posting messages.
So I looked for others of cheap traffic and audience source. Search "github python thumblr, twitter, pinterest, " shows many projects which are worth to test. I will write about them later.
Firstly, you have to collect much information of Facebook Groups - do not their purpose, members numbers, policies, if they are administered or not. To be profitable to you have to post to free groups, with high members numbers, to keep policies. To administer I programmed systematization tool and integrate data with pandas for sorting and update. Systematization program tool assigns members numbers to link and rules as well, therefore you can rearrange within one minute scraped from Facebook member profile Groups data to list - group name, members numbers and rules; and afterwards match with links and names while merging files. The program is great process hundreds of targeted members data and get best plan for marketing with highest visibility at low costs.
Secondly, you have to post preferably up to 20 post and make pauses. So you have to delete first twenty groups. Manual job is required. There are tasks that can be done - collection and updating of moderated, approval, disabled for posting groups. It will help to save postings to improve visibility to potential customers and improve SEO and overall conversions of online business. These tasks have been automated with programs. Thirdly, you should often update your posting messages.
So I looked for others of cheap traffic and audience source. Search "github python thumblr, twitter, pinterest, " shows many projects which are worth to test. I will write about them later.
Sunday, November 19, 2017
Different text standards and encoding methods
While programming and making test you may have perfectly performing programs. But if you add data from various sources pro use libraries like pandas or other which encode or decode text, your programs using text file data may may not perform tasks as programmed. One problem I encountered that pohmeliy.com bot does not read see text data urls which passed through pandas encoding interpreter which in turn enodes everything to UTF-8, which I solved by downloading list using pohmeliy.com tolist bot. And another problem some url lines attaches each to other hindering programmed performance or making list of urls unreadable. Encoding and encoding methods detection is universal problem. It is useful to read for understanding What Every Programmer Absolutely, Positively Needs To Know About Encodings And Character Sets To Work With Text to understand how text works with encodings. To detect text encodings I use python chardect
Chardet: The Universal Character Encoding Detector which detects some encodings standards and some variants with detection confidenc. Today I tested I got:
C:\Users\ANTRAS>chardetect C:\Pohmeliy_FB\FB_post_to_groupsCR\lists\CRGroups.txt C:\Pohmeliy_FB\FB_post_to_groupsSILK\lists\SILKGroups.txt
C:\Pohmeliy_FB\FB_post_to_groupsCR\lists\CRGroups.txt: ascii with confidence 1.0
C:\Pohmeliy_FB\FB_post_to_groupsSILK\lists\SILKGroups.txt: ascii with confidence 1.0
So you you have to find one data source or decode and encode to same byte order variant. About Byte order mark you can read on Wikipedia.org .
Chardet: The Universal Character Encoding Detector which detects some encodings standards and some variants with detection confidenc. Today I tested I got:
C:\Users\ANTRAS>chardetect C:\Pohmeliy_FB\FB_post_to_groupsCR\lists\CRGroups.txt C:\Pohmeliy_FB\FB_post_to_groupsSILK\lists\SILKGroups.txt
C:\Pohmeliy_FB\FB_post_to_groupsCR\lists\CRGroups.txt: ascii with confidence 1.0
C:\Pohmeliy_FB\FB_post_to_groupsSILK\lists\SILKGroups.txt: ascii with confidence 1.0
So you you have to find one data source or decode and encode to same byte order variant. About Byte order mark you can read on Wikipedia.org .
Tuesday, November 14, 2017
PYAHK AutoHotKey via Python
AutoHotKey is a powerful task automator with a terrible scripting language built-in. Python is a powerful scripting language with old, unmaintained, and incomplete automation modules. Combining the two creates a powerful automation tool with a powerful scripting back-end with access to all the power of the Python standard library.
This is made possible by the amazing Python ctypes module and the AutoHotKey.dll project. Together they allow exchange of data between both scripting engines, and the execution of AHK commands from Python.
link PYAHK AutoHotKey via Python
Combining with pywinauto and SWAPY automation tool and PYAHK AutoHotKey via Python automated various software solutions can be programmed. Before testing I assume that cycle can be programmed using iterate range function, but for sound I have idea to analysis record speech and base iteration number to achieve necessary physics.
This is made possible by the amazing Python ctypes module and the AutoHotKey.dll project. Together they allow exchange of data between both scripting engines, and the execution of AHK commands from Python.
link PYAHK AutoHotKey via Python
Combining with pywinauto and SWAPY automation tool and PYAHK AutoHotKey via Python automated various software solutions can be programmed. Before testing I assume that cycle can be programmed using iterate range function, but for sound I have idea to analysis record speech and base iteration number to achieve necessary physics.
Sunday, November 12, 2017
Scrapy Crawling projects Search Github python scrapy
Useful projects for scrapy. I will test or use them for coding my tailor made. These crawlers do not include some pypi projects llike imagebot.
Today I code Pandas data import and updating and sorting by numbers as well strings and dropping repetitive lines. The day was successful. To be more productive I will program scrapy codes to collect information for freelancing.
Scrapy, a fast high-level web crawling & scraping framework for Python.
GitHub - geekan/scrapy-examples: Multifarious Scrapy examples .
https://github.com/geekan/scrapy-examples
scrapy-examples - Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.
GitHub - scrapy/dirbot: Scrapy project to scrape public web directories.
https://github.com/scrapy/dirbot
dirbot - Scrapy project to scrape public web directories (educational) [DEPRECATED]
GitHub - scrapy/quotesbot: Quotebot for one bot.
https://github.com/scrapy/quotesbot
This is a Scrapy project to scrape quotes from famous people from http://quotes.toscrape.com (github repo). This project is only meant for educational purposes.
GitHub - rmax/scrapy-redis: Redis-based components for Scrapy.
https://github.com/rmax/scrapy-redis
Redis-based components for Scrapy. Free software: MIT license; Documentation: https://scrapy-redis.readthedocs.org.
GitHub - scrapy/scrapely: A pure-python HTML screen-scraping library
https://github.com/scrapy/scrapely
A pure-python HTML screen-scraping library. Contribute to scrapely development by creating an account on GitHub.
GitHub - mjhea0/Scrapy-Samples: Scrapy examples crawling Craigslist
https://github.com/mjhea0/Scrapy-Samples
Scrapy examples crawling Craigslist. Contribute to Scrapy-Samples development by creating an account on GitHub.
GitHub - scrapinghub/portia: Visual scraping for Scrapy
https://github.com/scrapinghub/portia
Visual scraping for Scrapy. Contribute to portia development by creating an account on GitHub.
GitHub - edx/pa11ycrawler: Python crawler (using Scrapy) that uses.
https://github.com/edx/pa11ycrawler
pa11ycrawler - Python crawler (using Scrapy) that uses Pa11y to check accessibility of pages as it crawls.
GitHub - eloyz/reddit: .
https://github.com/eloyz/reddit
2015-02-05 - Scrapy (Python Framework) Example using reddit.com.
GitHub - vinta/BlackWidow: Web crawler using Scrapy
https://github.com/vinta/BlackWidow
Web crawler using Scrapy http://heelsfetishism.com. Install. $ sudo apt-get install python-dev libxml2-dev libxslt1-dev $ pip install -r requirements.txt.
GitHub - istresearch/scrapy-cluster: This Scrapy project uses Redis.
https://github.com/istresearch/scrapy-cluster
scrapy-cluster - This Scrapy project uses Redis and Kafka.
GitHub - scrapy/w3lib: Python library of web-related functions
https://github.com/scrapy/w3lib
Python library of web-related functions. Contribute to w3lib development by creating an account on GitHub.
GitHub - scrapy-plugins/scrapy-deltafetch: Scrapy spider middleware.
https://github.com/scrapy-plugins/scrapy-deltafetch
scrapy-deltafetch - Scrapy spider middleware to ignore requests to pages containing items ... DeltaFetch middleware depends on Python's bsddb3 package.
GitHub - scrapy/scrapyd: A service daemon to run Scrapy spiders
https://github.com/scrapy/scrapyd
A service daemon to run Scrapy spiders. Scrapyd is a service for running Scrapy spiders.
Scrapy Plugins · GitHub
https://github.com/scrapy-plugins
Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls.
Web Scraping with Scrapy and MongoDB - Real Python
https://realpython.com/.../python/web-scraping-with-scrapy-and-mo...
Deploy your Scrapy Spiders from GitHub – The Scrapinghub Blog
https://blog.scrapinghub.com/.../deploy-your-scrapy-spiders-from-gi...
2017-04-19 - Up until now, your deployment process using Scrapy Cloud has probably ...
Scrapy Cloud's new GitHub integration will help you ensure that your.
python - Scrapy and github login - Stack Overflow
https://stackoverflow.com/questions/.../scrapy-and-github-login
2016-11-26 - You shall try like this def parse(self, response): print "in parse function" yield FormRequest.from_response( response, ...
Running scrapy spider programmatically - Musings of a programmer
https://kirankoduru.github.io/python/running-scrapy-programmatica.
Please check the project on github. The Scrapy Spider : It is a python class in the scrapy framework that is responsible for fetching URLs and parsing the
scrapy-crawlera 1.2.4 : Python Package Index
https://pypi.python.org/pypi/scrapy-crawlera
Crawlera middleware for Scrapy. scrapy-crawlera 1.2.4 .Author: Raul Gallegos; Home Page: https://github.com/scrapy-plugins/scrapy-crawlera;
Webscraping Airbnb with scrapy – - Latest Posts
www.verginer.eu/blog/web-scraping-airbnb/
You can find the complete code here as github repo, feel free to fork, clone.
Scrapy Tutorial: Web Scraping Craigslist – Web Scraping with Python
python.gotrained.com/scrapy-tutorial-web-scraping-craigslist/
Craigslist Scrapy Tutorial on GitHub - You can also find all the spiders we explained in this Python Scrapy tutorial on GitHub.
scrapy-tdd 0.1.3 : Python Package Index - PyPI https://pypi.python.org/pypi/scrapy-tdd/0.1.3 Helpers and examples to build Scrapy Crawlers in a test driven way.
The project to be tested firstly.
For pypi scrapy crawlers I used search "pypi scrapy crawlers".Today I code Pandas data import and updating and sorting by numbers as well strings and dropping repetitive lines. The day was successful. To be more productive I will program scrapy codes to collect information for freelancing.
Scrapy, a fast high-level web crawling & scraping framework for Python.
GitHub - geekan/scrapy-examples: Multifarious Scrapy examples .
https://github.com/geekan/scrapy-examples
scrapy-examples - Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.
GitHub - scrapy/dirbot: Scrapy project to scrape public web directories.
https://github.com/scrapy/dirbot
dirbot - Scrapy project to scrape public web directories (educational) [DEPRECATED]
GitHub - scrapy/quotesbot: Quotebot for one bot.
https://github.com/scrapy/quotesbot
This is a Scrapy project to scrape quotes from famous people from http://quotes.toscrape.com (github repo). This project is only meant for educational purposes.
GitHub - rmax/scrapy-redis: Redis-based components for Scrapy.
https://github.com/rmax/scrapy-redis
Redis-based components for Scrapy. Free software: MIT license; Documentation: https://scrapy-redis.readthedocs.org.
GitHub - scrapy/scrapely: A pure-python HTML screen-scraping library
https://github.com/scrapy/scrapely
A pure-python HTML screen-scraping library. Contribute to scrapely development by creating an account on GitHub.
GitHub - mjhea0/Scrapy-Samples: Scrapy examples crawling Craigslist
https://github.com/mjhea0/Scrapy-Samples
Scrapy examples crawling Craigslist. Contribute to Scrapy-Samples development by creating an account on GitHub.
GitHub - scrapinghub/portia: Visual scraping for Scrapy
https://github.com/scrapinghub/portia
Visual scraping for Scrapy. Contribute to portia development by creating an account on GitHub.
GitHub - edx/pa11ycrawler: Python crawler (using Scrapy) that uses.
https://github.com/edx/pa11ycrawler
pa11ycrawler - Python crawler (using Scrapy) that uses Pa11y to check accessibility of pages as it crawls.
GitHub - eloyz/reddit: .
https://github.com/eloyz/reddit
2015-02-05 - Scrapy (Python Framework) Example using reddit.com.
GitHub - vinta/BlackWidow: Web crawler using Scrapy
https://github.com/vinta/BlackWidow
Web crawler using Scrapy http://heelsfetishism.com. Install. $ sudo apt-get install python-dev libxml2-dev libxslt1-dev $ pip install -r requirements.txt.
GitHub - istresearch/scrapy-cluster: This Scrapy project uses Redis.
https://github.com/istresearch/scrapy-cluster
scrapy-cluster - This Scrapy project uses Redis and Kafka.
GitHub - scrapy/w3lib: Python library of web-related functions
https://github.com/scrapy/w3lib
Python library of web-related functions. Contribute to w3lib development by creating an account on GitHub.
GitHub - scrapy-plugins/scrapy-deltafetch: Scrapy spider middleware.
https://github.com/scrapy-plugins/scrapy-deltafetch
scrapy-deltafetch - Scrapy spider middleware to ignore requests to pages containing items ... DeltaFetch middleware depends on Python's bsddb3 package.
GitHub - scrapy/scrapyd: A service daemon to run Scrapy spiders
https://github.com/scrapy/scrapyd
A service daemon to run Scrapy spiders. Scrapyd is a service for running Scrapy spiders.
Scrapy Plugins · GitHub
https://github.com/scrapy-plugins
Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls.
Web Scraping with Scrapy and MongoDB - Real Python
https://realpython.com/.../python/web-scraping-with-scrapy-and-mo...
Deploy your Scrapy Spiders from GitHub – The Scrapinghub Blog
https://blog.scrapinghub.com/.../deploy-your-scrapy-spiders-from-gi...
2017-04-19 - Up until now, your deployment process using Scrapy Cloud has probably ...
Scrapy Cloud's new GitHub integration will help you ensure that your.
python - Scrapy and github login - Stack Overflow
https://stackoverflow.com/questions/.../scrapy-and-github-login
2016-11-26 - You shall try like this def parse(self, response): print "in parse function" yield FormRequest.from_response( response, ...
Running scrapy spider programmatically - Musings of a programmer
https://kirankoduru.github.io/python/running-scrapy-programmatica.
Please check the project on github. The Scrapy Spider : It is a python class in the scrapy framework that is responsible for fetching URLs and parsing the
scrapy-crawlera 1.2.4 : Python Package Index
https://pypi.python.org/pypi/scrapy-crawlera
Crawlera middleware for Scrapy. scrapy-crawlera 1.2.4 .Author: Raul Gallegos; Home Page: https://github.com/scrapy-plugins/scrapy-crawlera;
Webscraping Airbnb with scrapy – - Latest Posts
www.verginer.eu/blog/web-scraping-airbnb/
You can find the complete code here as github repo, feel free to fork, clone.
Scrapy Tutorial: Web Scraping Craigslist – Web Scraping with Python
python.gotrained.com/scrapy-tutorial-web-scraping-craigslist/
Craigslist Scrapy Tutorial on GitHub - You can also find all the spiders we explained in this Python Scrapy tutorial on GitHub.
Saturday, November 11, 2017
How do I find and remove duplicate rows in pandas?
Great tutorial how to find duplicated rows and remove them either identical or by column. Use drop_dublicates plus column set plus shape methods. See video.
Python automation blog Python scrapy examples
https://simply-python.com/tag/scrapy/
Simply Blog seems useful with example for Automation. After I finish Pandas automation of data, posting administration project and integrated SFS Win sound engineering automated workflow management.
Friday, November 10, 2017
Pandas Data frame update update
Pandas data frame update should be one of the most useful functions.
DataFrame.update(other, join='left', overwrite=True, filter_func=None, raise_conflict=False)[source]¶
Modify DataFrame in place using non-NA values from passed DataFrame. Aligns on indices
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.update.html
Pandas dataframe merge with update data
Pandas dataframe merge with update data
print (pd.concat([df1, df2]).drop_duplicates('date', keep='last')) date value0 2017-01-01 11 2017-01-02 12 2017-01-03 10 2017-01-04 21 2017-01-05 22 2017-01-06 23 2017-01-07 24 2017-01-08 2
Concat works, but drop_dublicates('ID', keep='last') does not.
Wednesday, November 8, 2017
Python scripts for search engines.
https://pythonexample.com/
Search engine scripts, some are really useful. To be tested. And Copernicus for download information from social media. OSINT - Open Source Intelligence "Leading Intelligence and Investigation Technique"
New! Automatically Discover Website Connections Through Tracking Codes
http://www.automatingosint.com/blog/2017/07/osint-website-connections-tracking-codes/
Python examples Timeline
 | Title | Developer | Keyword | Description | Date |
|---|---|---|---|---|---|
 | serializers.py | clearinterface | serializers | 2017-11-08 | |
| python_ftp.py | clearinterface | python | python ftp code | 2017-11-08 |
| api.py | clearinterface | api.py - p=126 | 2017-11-08 | |
| http_request.py | nhrwtr | http request | HTTPの勉強 | 2017-11-08 |
| autowihte | laamalif | autowihte | auto white correction opencv | 2017-11-08 |
| Queue.py | bosskun | queue | Queue | 2017-11-08 |
| PyEx1.py | Testosterpwn | pyex1 | PracticePython.org Exercise 1 + bonus objectives | 2017-11-08 |
| DoublyLinkedList.py | bosskun | doublylinkedlist | DoublyLinkedList | 2017-11-08 |
| ghost_in_the_heap.py | peternguyen93 | ghost heap | 2017-11-08 | |
| dummy-web-server.py | sherlockwu | dummy server | a minimal http server in python. Responds to GET, HEAD, POST requests, but will fail on anything else. | 2017-11-08 |
| PyEx2.py | Testosterpwn | pyex2 | PracticePython.org Exercise 2 + bonus objectives | 2017-11-08 |
| gcloud_bq_tables.py | krnr | gcloud tables | example of BigQuery tables creation | 2017-11-08 |
| python-get-count-by-date.py | dgadiraju | python count date | 2017-11-08 | |
| ytdl.pyw | DimChtz | ytdl | Python script (GUI) to download YouTube videos (as video or audio). | 2017-11-08 |
| guillet.py | dcw32 | guillet | Guillet Data | 2017-11-08 |
| focus_group.py | Sumenia | focus group | focus group | 2017-11-08 |
| genie.py | minhtt159 | genie | SvATTT2017 - Crypto2 | 2017-11-08 |
| combine_images_vertically.py | daite | combine images vertically | combine_images_vertically | 2017-11-08 |
| jsreport.py | rylincoln | jsreport | python jsreport batch | 2017-11-08 |
| pair_checker.py | chewingF | pair checker | 2017-11-08 |
Sunday, November 5, 2017
Useful sleepless night because of neighbors night became useful since I systemized Python source ideas
Because of noise and moved furniture in neighbor flat and felt stress and though about projects if I have sleepless nights. Psychologist assert that minor short-term stress are useful for innovative or creative ideas. I was concerned about organization capabilities. Python has more than 100 thousand projects and libraries published, "do not invent bicycles" is applicable.
Main ideas to write new code:
1. Systemize your ideas codes, i.e. my tree of codes for functions. For examples - copied or developed codes for data processing, ext file automation and etc. Base system on ready to integrate solution codes and vice versa on specific narrow (for example string operation) with necessary description how adapt and develop code. For this purpose draft tree for python codes folders structures.
2. Secondly I download python books (O'Reily free python books, Python Cookbook, Python Essential Reference and etc.. Systemize book and write description for solutions. Will it be faster to find with relevant searches on stackoverflow, online tutorials, code search engines? I do not know. I just know that you will grasp ideas of may be several or numerous application possibilities, or may be alternative and write code search wording using technical term.
3. Exploit with priority pypi and github, to find projects or copy necessary part of codes. Systemize sources of urls with bookmarks. Programming is dynamic, the code within one year can become obselete.
4. Systemize specific libraries sources and tutorials for specific needs and exploit.
The system is needed to automate tasks, jobs; so I look for ways to streamline/automate code creations. Also I can reduce time needed to learn programming and find solutions.
So on github I found projects for codes to save time.
Main ideas to write new code:
1. Systemize your ideas codes, i.e. my tree of codes for functions. For examples - copied or developed codes for data processing, ext file automation and etc. Base system on ready to integrate solution codes and vice versa on specific narrow (for example string operation) with necessary description how adapt and develop code. For this purpose draft tree for python codes folders structures.
2. Secondly I download python books (O'Reily free python books, Python Cookbook, Python Essential Reference and etc.. Systemize book and write description for solutions. Will it be faster to find with relevant searches on stackoverflow, online tutorials, code search engines? I do not know. I just know that you will grasp ideas of may be several or numerous application possibilities, or may be alternative and write code search wording using technical term.
3. Exploit with priority pypi and github, to find projects or copy necessary part of codes. Systemize sources of urls with bookmarks. Programming is dynamic, the code within one year can become obselete.
4. Systemize specific libraries sources and tutorials for specific needs and exploit.
The system is needed to automate tasks, jobs; so I look for ways to streamline/automate code creations. Also I can reduce time needed to learn programming and find solutions.
So on github I found projects for codes to save time.
- Project 1 - E-mail and phone number extractor
- Project 2 - Strong Password Detection
- Project 3 - Regular expression version for strip()
- Project 4 - Multiclipboard
- Project 5 - Mad Libs
- Project 6 - Regex Search()
- Project 7 - Renaming the files
- Project 8 - Backup a file/folder into a zip file
- Project 9 - Map Me!
- Project 10 - Feeling Lucky Today!
- Project 11 - 2048!
- Project 12 - Combining pages from PDFs
- project 13 - Copying '.jpg' and '.pdf' files into a newfolder
- Project 14 - Deleting Uneeded Files from PC
- Project 15 - Brute-Force PDF Password Breaker
- Project 16 - UNIX Password Cracker
- Project 17 - ZIP Password Cracker
- EASTER EGG FUN - Try it for fun!
- Profanity check - Save your A$$!
- Pirate speech - Translate into a pirate
- Count Frequency - Counts the frequency of each word in a text file.
6 Useful Python Libraries Recommended by
https://www.codementor.io/mjpieters/6-useful-python-libraries-recommended-by-1-stack-overflow-answerer-aid85efxi
New libraries for automation:
https://www.codementor.io/mjpieters/6-useful-python-libraries-recommended-by-1-stack-overflow-answerer-aid85efxi
New libraries for automation:
RoboBrowser: Your friendly neighborhood web scraper
Requests: HTTP for Humans Library to read and test/
Collection / Learn Python List of sources to read, test.
Collection /
https://www.codementor.io/collections/learn-python-bwbc63ulz
Unexpectedly I found searching solutions https://ugoproto.github.io/ugo_py_doc/ Python documentation file with cheatsheets. This site contains cheatsheets, commands summaries and other helpful information to rehearse. I stumbled to problem how to read text file with string.lsplit method (and rstrtip and so on). This kind of text files manipulation is necessary to prepare data for Pandas frame since because of text markups I get undesirable underscore beginning first line character. So I though about systemizing file read and after processing write methods. I use at least three read writing to methods and rewriting. Also parsing (reading) or looping through lines methods. And why they differ, when to use them since there are many methods with different parameters requirements and modules (libraries) with own environments (codings). This help to streamline automation of data processing and writing out results. For example regex library may require one method, python commands another. Most popular libraries preferbaly be systemized.
Youtube.com look for:
Processing Text in Python
Parsing text files in Python
Reading text files in Python
Looping text files in Python
Unexpectedly I found searching solutions https://ugoproto.github.io/ugo_py_doc/ Python documentation file with cheatsheets. This site contains cheatsheets, commands summaries and other helpful information to rehearse. I stumbled to problem how to read text file with string.lsplit method (and rstrtip and so on). This kind of text files manipulation is necessary to prepare data for Pandas frame since because of text markups I get undesirable underscore beginning first line character. So I though about systemizing file read and after processing write methods. I use at least three read writing to methods and rewriting. Also parsing (reading) or looping through lines methods. And why they differ, when to use them since there are many methods with different parameters requirements and modules (libraries) with own environments (codings). This help to streamline automation of data processing and writing out results. For example regex library may require one method, python commands another. Most popular libraries preferbaly be systemized.
Youtube.com look for:
Processing Text in Python
Parsing text files in Python
Reading text files in Python
Looping text files in Python
Thursday, November 2, 2017
Pytrends for small business by General Mills Minneapolis
On github I found tools pytrends for small business to use Google trends developed by General Mills Minneapolis, MN http://generalmills.com/ (Big Name).
https://github.com/GeneralMills/pytrends#interest-over-time
Unofficial API for Google Trends
Allows simple interface for automating downloading of reports from Google Trends. Main feature is to allow the script to login to Google on your behalf to enable a higher rate limit. Only good until Google changes their backend again :-P. When that happens feel free to contribute!
The following API methods are available:
Interest Over Time: returns historical, indexed data for when the keyword was searched most as shown on Google Trends' Interest Over Time section.
Interest by Region: returns data for where the keyword is most searched as shown on Google Trends' Interest by Region section.
Related Topics: returns data for the related keywords to a provided keyword shown on Google Trends' Related Topics section.
Related Queries: returns data for the related keywords to a provided keyword shown on Google Trends' Related Queries section.
Trending Searches: returns data for latest trending searches shown on Google Trends' Trending Searches section.
Top Charts: returns the data for a given topic shown in Google Trends' Top Charts section.
Suggestions: returns a list of additional suggested keywords that can be used to refine a trend search.
With red is report dates that I cannot change, I gues it is 5 years period start.
C:\Users\ANTRAS\Desktop> python GOOGLETRENDSAMPLE.py
pizza bagel isPartial
date
2012-11-04 56 1 False
2012-11-11 55 1 False
2012-11-18 54 1 False
2012-11-25 56 1 False
2012-12-02 57 1 False
pizza bagel
geoName
Argentina 19 0
Australia 69 1
Austria 25 0
Belgium 34 0
Brazil 24 0
{}
date exploreUrl \
0 20171101 /trends/explore#q=Academy&date=now+7-d&geo=US
1 20171101 /trends/explore#q=brian+hoyer&date=now+7-d&geo=US
2 20171101 /trends/explore#q=d%C3%ADa+de+muertos&date=now...
3 20171101 /trends/explore#q=Donna+Brazile&date=now+7-d&g...
4 20171101 /trends/explore#q=spectrum+internet&date=now+7...
formattedTraffic hotnessColor hotnessLevel \
0 Daugiau nei 100 000 #e68033 3.0
1 Daugiau nei 50 000 #f0a049 2.0
2 Daugiau nei 50 000 #f0a049 2.0
3 Daugiau nei 50 000 #f0a049 2.0
4 Daugiau nei 50 000 #f0a049 2.0
imgLinkUrl \
0 http://www.ajc.com/news/traffic/breaking-offic...
1 http://www.espn.com/blog/new-england-patriots/...
2 https://missionlocal.org/2017/11/the-sweet-bre...
3 https://www.huffingtonpost.com/entry/donna-bra...
4 http://www.statesman.com/news/local/spectrum-c...
imgSource \
0 Atlanta Journal Constitution
1 ESPN (blog)
2 Mission Local
3 HuffPost
4 Austin American-Statesman
imgUrl \
0 //t0.gstatic.com/images?q=tbn:ANd9GcTUzhTCxnzH...
1 //t1.gstatic.com/images?q=tbn:ANd9GcSS71j5J2s4...
2 //t3.gstatic.com/images?q=tbn:ANd9GcTQs9Ec34PR...
3 //t3.gstatic.com/images?q=tbn:ANd9GcR-n6eWjyiV...
4 //t0.gstatic.com/images?q=tbn:ANd9GcRR11HDNfBU...
newsArticlesList relatedSearchesList \
0 [{u'snippet': u'An officer was hit early Thurs... []
1 [{u'snippet': u'As part of the 2019 salary, <b... []
2 [{u'snippet': u'All month, residents and touri... []
3 [{u'snippet': u'<b>Donna Brazile</b>, former i... []
4 [{u'snippet': u'Earlier: <b>Spectrum</b> custo... []
safe shareUrl startTime \
0 1.0 https://www.google.com/trends/hottrends?stt=Ac... 1.509628e+09
1 1.0 https://www.google.com/trends/hottrends?stt=Br... 1.509624e+09
2 1.0 https://www.google.com/trends/hottrends?stt=D%... 1.509631e+09
3 1.0 https://www.google.com/trends/hottrends?stt=Do... 1.509638e+09
4 1.0 https://www.google.com/trends/hottrends?stt=Sp... 1.509624e+09
title titleLinkUrl \
0 Academy //www.google.com/search?q=Academy
1 Brian Hoyer //www.google.com/search?q=Brian+Hoyer
2 Día de Muertos //www.google.com/search?q=D%C3%ADa+de+Muertos
3 Donna Brazile //www.google.com/search?q=Donna+Brazile
4 Spectrum internet //www.google.com/search?q=Spectrum+internet
trafficBucketLowerBound
0 100000.0
1 50000.0
2 50000.0
3 50000.0
4 50000.0
delta deltaSummary \
0 3.0 <b> + 3</b> nuo pra?jusio m?nesio
1 1.0 <b> + 1</b> nuo pra?jusio m?nesio
2 2.0 <b> + 2</b> nuo pra?jusio m?nesio
3 NaN <b>Nauja</b> diagramoje
4 -3.0 <b>-3</b> nuo pra?jusio m?nesio
description \
0 {u'source': u'Wikipedia', u'sourceUrl': u'http...
1 {u'source': u'Wikipedia', u'sourceUrl': u'http...
2 {u'source': u'Wikipedia', u'sourceUrl': u'http...
3 {u'source': u'Wikipedia', u'sourceUrl': u'http...
4 {u'source': u'Wikipedia', u'sourceUrl': u'http...
exploreUrl hotnessColor \
0 /trends/explore?q=/m/09gkdy4&date=2016-11-1+20... #d04108
1 /trends/explore?q=/m/0214xp&date=2016-11-1+201... #d04108
2 /trends/explore?q=/m/0gs6vr&date=2016-11-1+201... #d04108
3 /trends/explore?q=/m/01xwv7&date=2016-11-1+201... #d04108
4 /trends/explore?q=/m/0bdxs5&date=2016-11-1+201... #d04108
hotnessLevel idForTracking isNew peakRankSummary \
0 5 Ariana Grande False Pakilo iki <b>#1</b> (per m?nes?)
1 5 Sunny Leone False Pakilo iki <b>#2</b> (per m?nes?)
2 5 Selena Gomez False Pakilo iki <b>#1</b> (per m?nes?)
3 5 Dave Chappelle True Pakilo iki <b>#3</b> (per m?nes?)
4 5 Miley Cyrus False Pakilo iki <b>#1</b> (per m?nes?)
shareUrl \
0 https://trends.google.com/trends/topcharts?vm=...
1 https://trends.google.com/trends/topcharts?vm=...
2 https://trends.google.com/trends/topcharts?vm=...
3 https://trends.google.com/trends/topcharts?vm=...
4 https://trends.google.com/trends/topcharts?vm=...
timePeriodsInChartSummary title \
0 <b>51 m?nuo</b> 10 populiariausi? diagramoje Ariana Grande
1 <b>17 m?nesi?</b> 10 populiariausi? diagramoje Sunny Leone
2 <b>105 m?nesiai</b> 10 populiariausi? diagramoje Selena Gomez
3 <b>9 m?nesiai</b> 10 populiariausi? diagramoje Dave Chappelle
4 <b>121 m?nuo</b> 10 populiariausi? diagramoje Miley Cyrus
titleLength titleLinkUrl trafficSummary \
0 13 //www.google.com/search?q=Ariana+Grande 100
1 11 //www.google.com/search?q=Sunny+Leone 84
2 12 //www.google.com/search?q=Selena+Gomez 81
3 14 //www.google.com/search?q=Dave+Chappelle 81
4 11 //www.google.com/search?q=Miley+Cyrus 80
twitterShareUrlTitle
0 Ariana Grande yra Nr. 1 ?Google? tendencij? di...
1 Sunny Leone yra Nr. 2 ?Google? tendencij? diag...
2 Selena Gomez yra Nr. 3 ?Google? tendencij? dia...
3 Dave Chappelle yra Nr. 4 ?Google? tendencij? d...
4 Miley Cyrus yra Nr. 5 ?Google? tendencij? diag...
[{u'type': u'Dish', u'mid': u'/m/0663v', u'title': u'Pizza'}, {u'type': u'Restaurant company', u'mid': u'/m/09cfq', u'title': u'Pizza Hut'}, {u'type': u'Topic', u'mid': u'/m/05m1lt', u'title': u'Pizza delivery'}, {u'type': u'Restaurant company', u'mid': u'/m/03clwm', u'title': u"Domino's Pizza"}, {u'type': u'Fast food restaurant company', u'mid': u'/m/021sqj', u'title': u'Pizza Pizza'}]
So I tested my keywords to find best images SEO possibilities in England Norther town. green lines show best search traffic concentrate (competitiveness aside). This tool increase productivity after all, since you can also search Youtube, shoppers site also.
C:\Users\ANTRAS\Desktop> python GOOGLETRENDENGLAND.py
flowers delivery Manchester flowers Manchester \
date
2012-11-04 0 0
2012-11-11 0 100
2012-11-18 0 0
2012-11-25 0 0
2012-12-02 0 0
send flowers Manchester flowers of Manchester \
date
2012-11-04 0 0
2012-11-11 0 0
2012-11-18 0 0
2012-11-25 0 0
2012-12-02 0 0
flowers in Manchester isPartial
date
2012-11-04 0 False
2012-11-11 0 False
2012-11-18 0 False
2012-11-25 0 False
2012-12-02 0 False
flowers delivery Manchester flowers Manchester \
geoName
England 15 100
send flowers Manchester flowers of Manchester flowers in Manchester
geoName
England 0 8 31
{}
Subscribe to:
Posts (Atom)