Michael Yin

In May, 2018, I am hired as a remote developer by Torchbox (Creator of Wagtail CMS)

I am responsible to build web projects based on Wagtail CMS.

The biggest feature in this project is that we need to build a multiform system for the client, the form fields on form page need to be editable (no coding required). And some form fields have some custom features

We implement this feature based on StreamField and FormBuilder of Wagtail, and we call it StreamForm. We also write some javascript to increase the user experience. Clients are very satisfied with it.

My Guest Post On Wagtal's official site. Add a Wagtail blog app to your Django project

StyleSage is a big data company in fashion industry, they need to collect fashion data from dozens of e-commerce websites such as hm, Zara and etc.

To collect huge data in a quick and efficient way, we build our web scraping system based on Scrapy framework.

Because we do not rely on the headless browser so our scraping speed can be guaranteed. At the same time, we build and maintain proxy pool, backend no-sql DB to save, pre-process scraped data.

I mainly solved these problems in this project.

1. Analyze web requests and implement in Scrapy.
2. Analyze javascript code & JSON code to extract data.
3. Avoid spiders being banned by sending too many requests.
4. How to quickly review data by writing custom pipeline and some stats tool such as Pandas.

ScrapingClub is my side project that helps newbie developers to learn web scraping in a better way.

Most blog posts about Web Scraping talks about how to crawl a specific website or pages, but I insist it makes more sense to help people to learn how to analyze the website and choose the right way to get the job done well.

That is why I build ScrapingClub, my goal is to try to break down a complex web scraping mission such as crawling a bunch of websites to some small tasks so people can learn how to solve them step by step.

Right now it includes 10 practical web scraping exercises already.

Wagtail Bootstrap Blog is developed exclusively for Wagtail Tutorial Build Blog With Wagtail CMS, which shows people how to create a Wagtail blog using Bootstrap step by step. You can also import it into your Django project to quickly add professional blog function based on Wagtail.

I wish this open source project can help people to learn Wagtail CMS, which is a very powerful framework in Django ecosystem.