souvenir
Tag cloud
Picture wall
Daily
RSS Feed
  • RSS Feed
  • Daily Feed
Filters

Links per page

  • 20 links
  • 50 links
  • 100 links

Filters

Untagged links
8 results tagged scraper  ✕   ✕
ScrapeGraphAI https://github.com/ScrapeGraphAI/Scrapegraph-ai
10/12/2024 cluster icon
  • Scrapy : Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be...
  • mock : A Python Mocking and Patching Library for Testing
  • pyo : pyo is a Python module written in C to help digital signal processing script creation.
  • pydantic : Data validation and settings management using python type annotations.
  • TextBlob : TextBlob is a Python (2 and 3) library for processing textual data. It provides a simple API for diving into common natural language processing (NLP) ...

ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.).

python scraper library llm
Colly http://go-colly.org/
24/05/2021 cluster icon
  • Beego : An open source framework to build and develop your applications in the Go way
  • Goutte : Goutte is a screen scraping and web crawling library for PHP. Goutte provides a nice API to crawl websites and extract data from the HTML/XML response...
  • Scrapy : Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be...
  • Ponzu : Ponzu is a powerful and efficient open-source HTTP server framework and CMS. It provides automatic, free, and secure HTTP/2 over TLS (certificates obt...
  • Fiber : Fiber is an Express inspired web framework build on top of Fasthttp, the fastest HTTP engine for Go. Designed to ease things up for fast development w...
thumbnail

Colly is a Go framework that provides a clean interface to write any kind of crawler/scraper/spider

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

go scraper crawler framework datamining archive
Scrapoxy https://scrapoxy.io/
23/06/2020 cluster icon
  • Envoy Proxy : Envoy is an L7 proxy and communication bus designed for large modern service oriented architectures.
  • Cool QL Cool : CoolQLCool (CQC) is an open source GraphQL server that allows you to turn websites into GraphQL APIs.
  • Portia : Portia is a tool that allows you to visually scrape websites without any programming knowledge required. With Portia you can annotate a web page to id...
  • gRPC : gRPC is a modern, open source remote procedure call (RPC) framework that can run anywhere. It enables client and server applications to communicate tr...
  • Botpress : Botpress is an open-source bot creation tool written in Javascript. It is powered by a rich set of open-source modules built by the community. We like...
thumbnail

Scrapoxy hides your webscraper behind a cloud.

It starts a pool of proxies to relay your requests.

Now, you can crawl without thinking about blacklisting!

opensource scraper proxy
Cool QL Cool https://coolql.cool/
30/12/2018 cluster icon
  • Appwrite : Appwrite is an end-to-end backend server for Web, Mobile, Native, or Backend apps packaged as a set of Docker microservices. Appwrite abstracts the co...
  • GraphQL : GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandabl...
  • Prisma : Prisma turns your database into a powerful GraphQL API which you can either use to build your own GraphQL server or connect to it directly from the fr...
  • Matrix : Matrix is an open standard for decentralised communication, providing simple HTTP APIs and open source reference implementations for securely distribu...
  • WireMock : Free and Open Source tool for building mock APIs. Create stable development environments, isolate yourself from flakey 3rd parties and simulate APIs t...

CoolQLCool (CQC) is an open source GraphQL server that allows you to turn websites into GraphQL APIs.

graphql scraper opensource server api
Portia https://github.com/scrapinghub/portia
21/02/2016 cluster icon
  • Caire : Caire is a content aware image resize library
  • cAdvisor : cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running containe...
  • Beekeeper Studio : Open Source cross-platform SQL Editor and Database Manager. Use Beekeeper Studio to query and manage your relational databases, like MySQL, Postgres, ...
  • unleash : Open-source feature management solution built for developers.
  • Cloud Carbon Footprint : Cloud Carbon Footprint is an open source tool that provides visibility and tooling to measure, monitor and reduce your cloud carbon emissions. Archite...
thumbnail

Portia is a tool that allows you to visually scrape websites without any programming knowledge required. With Portia you can annotate a web page to identify the data you wish to extract, and Portia will understand based on these annotations how to scrape data from similar pages.

scraper opensource tool
Upton https://github.com/propublica/upton
04/02/2014 cluster icon
  • Scrapy : Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be...
  • ScrapeGraphAI : ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML...
  • howler.js : howler.js is an audio library for the modern web. It defaults to Web Audio API and falls back to HTML5 Audio.
  • SHML · $(shell markup language) : SHML is a shell framework for faster and easier script development. HTML has CSS, Unix-like shells have ANSI/VT100 Control Sequences. SHML makes is ea...
  • Skeleton : Skeleton: A Dead Simple, Responsive CSS Boilerplate for Mobile-Friendly Development
thumbnail

Upton is a framework for easy web-scraping with a useful debug mode that doesn't hammer your target's servers. It does the repetitive parts of writing scrapers, so you only have to write the unique parts for each site.

scraper ruby library
Scrapy http://scrapy.org
27/09/2013 cluster icon
  • ScrapeGraphAI : ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML...
  • mock : A Python Mocking and Patching Library for Testing
  • Kartograph : The Kartograph map generator has just one method, which is there to generate SVG maps (surprise).
  • Essentia : Open-source library and tools for audio and music analysis, description and synthesis
  • Colly : Colly is a Go framework that provides a clean interface to write any kind of crawler/scraper/spider With Colly you can easily extract structured data ...

Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

python scraper crawler library
Goutte https://github.com/fabpot/Goutte
23/11/2011 cluster icon
  • Its Continuous! : The tweet-sized continuous integration app for PHP development.
  • PHP_CodeSniffer : PHP_CodeSniffer tokenises PHP, JavaScript and CSS files and detects violations of a defined set of coding standards.
  • GrumPHP : Sick and tired of defending code quality over and over again? GrumPHP will do it for you! This composer plugin will register some git hooks in your pa...
  • PHPMD - PHP Mess Detector : What PHPMD does is: It takes a given PHP source code base and look for several potential problems within that source. These problems can be things lik...
  • Chromeless : Chrome automation made simple. Runs locally or headless on AWS Lambda. Chromeless can be used to... Run 1000s of browser integration tests in paralle...
thumbnail

Goutte is a screen scraping and web crawling library for PHP.

Goutte provides a nice API to crawl websites and extract data from the HTML/XML responses.

php crawler test scraper
1642 links
Shaarli - The personal, minimalist, super-fast, database free, bookmarking service by the Shaarli community - Theme by kalvn