[SIP-130] Proposal for migrating from Mapbox to MapLibre
Motivation
With the Mapbox GL JS library becoming closed source and the existing map plugins available in Superset being based on the legacy data API, migrating to the MapLibre project (an open source Mapbox fork) should make the transition simpler while providing the opportunity to bring the map plugins into the fold of the new data framework.
Proposed Change
Create new map plugins using deck.gl (or perhaps kepler.gl?) and MapLibre GL JS (https://github.com/maplibre/maplibre-gl-js). Although backporting code (that is not covered by the BSD 3-Clause license) from Mapbox GL JS to MapLibre GL JS is not permitted, the expectation is that the gap in functions/features required by Superset to implement MapLibre is not large (see API documentation: https://maplibre.org/maplibre-gl-js/docs/).
The new plugins would be implemented using TypeScript and the new data API framework. Details on the specifics may have to be provided on a per plugin basis.
The MapLibre Style Specification (https://github.com/maplibre/maplibre-style-spec) may offer useful utilities for managing a MapLibre implementation.
Examples of what is possible with MapLibre GL JS are available here: https://maplibre.org/maplibre-gl-js/docs/examples/
New or Changed Public Interfaces
Plugins using Mapbox will need to be either modified, or (preferably) replaced. These will be discussed in subsequent SIPs.
Add new plugins/charts: - MapLibre - deckgl
Would deprecate: - legacy-plugin-chart-map-box (https://github.com/apache/superset/tree/master/superset-frontend/plugins/legacy-plugin-chart-map-box) - legacy-preset-chart-deckgl (https://github.com/apache/superset/tree/master/superset-frontend/plugins/legacy-preset-chart-deckgl)
New dependencies
MapLibre GL JS is licensed under the 3-Clause BSD license.
Migration Plan and Compatibility
It is unknown whether any database migrations are necessary. However, by deprecating the Mapbox-based plugins, existing charts based on those would have to be re-implemented by users or preferably supported by a migration.
Rejected Alternatives
Various past discussions covered alternatives such as kepler.gl (https://github.com/keplergl/kepler.gl), Leaflet (https://github.com/Leaflet/Leaflet), and a deck.gl + Leaflet implementation (https://github.com/zakjan/deck.gl-leaflet).
Although not all of these were firmly rejected, MapLibre potentially offers a less cumbersome migration.
Comment From: birkskyum
Disclaimer: i'm a board member in the maplibre non-profit org
Just a note on the alternatives - deck.gl works great with maplibre-gl, to the point that latest version of kepler.gl actually has migrated entirely from mapbox-gl to maplibre-gl, for the same licensing reasons as this issue point to.
It will be helpful when implementing a typescript superset map plugin for maplibre, that maplibre-gl, in contrast to mapbox-gl, is written entirely in typescript. I believe maplibre-gl currently has all the functionality required by superset, and it's for the most part a drop-in replacement - to that end, I'd be happy to help answer any question around maplibre-gl usage and technicalities around this migration.
Some other advantages that would apply here are:
advantages over mapbox-gl v1 or a fork of it
- The maintenance burden is moved to the MapLibre org. There is paid stable maintenance on the project, and also financial investments in new feature development, so PR's /issues won't be left for dead. All this is covered by the awesome MapLibre Sponsors who all benefit from the existence of MapLibre because it allow the cost to be shared instead of covering full maintenance/development of separate forks, and it keeps an industry wide compatibility.
- Lots of performance work has been done - notably microsoft has been helping out by profiling bing maps usage extensively on the millions of users there.
- maplibre has since v3 supported webgl2 which almost all users will be using at this point, and in the increasingly rare case it's not supported there is still a webgl1 fallback in place.
- there is a growing community around maplibre-gl, we've e.g. seen a 100% increase in downloads (now ~400k/weekly) just over the past half year.
- many companies are helping out with new features related to their respective domain of expertise.
- Continous support for emergent standards, like the new datasets from the Overture Maps Foundation as shown in this example.
here are contribution graphs, with the area approx. marked after the fork, to visually represent the scale of shared continuous investments which aren't present in mapbox-gl v1:
maplibre-gl ( 2000+ PR's with bug-fixes and improvements)
advantages over mapbox-gl v2+
maplibre-glandmapbox-gl v1are rendering libraries wheremapbox-gl v2is a rendering service. Both can be paired with data/styles from local files or services, but the difference is that by staying a renderer library, maplibre itself is free to use, and will run without an api key. This has multiple advantages:- allows rendering offline, on-prem
- gives better startup performance, because the code doesn't "calling home" to check the api key billing status
- It's more convenient for new users, not to have to signup for mapbox and get an api key.
- The maplibre license permits removing the logo from lower left, in case superset users rather see more of their plots.
- In case you bundle the renderer with superset, that something you can certainly do with maplibre-gl since it's just a library.
Comment From: rusackas
I would love to see this proposal / effort move forward. @Aalbedon do you want to move this forward with a [DISCUSS] thread on the Dev mailing list? Let me know if you need any assistance with that effort.
Comment From: rusackas
Checking in to see if you plan to implement this and/or carry the process through to completion, @Aalbedon. If not, this will eventually be closed due to inactivity.
Comment From: birkskyum
@rusackas , before closing it for inactivity, give me a ping. I don't mind championing the necessary work.
Comment From: rusackas
It won't be closed for quite a while... I just want to make sure it's able to move forward. There are really two parts to push on: 1) Making sure the process is followed (i.e. running this SIP through the Discuss and Vote threads on the mailing list) 2) Implementing the feature.
I can certainly assist with Item 1, but would only want to do so if someone's going to follow through here. If you're game, to tackle implementation @birkskyum, I'll happily start on the more bureaucratic part ;)
Comment From: birkskyum
That sounds great - I'm up for taking on the implementation part.
Comment From: rusackas
@birkskyum do you think we should limit the scope of this SIP to the Mabbox -> MapLibre migration? Then after that, we can address a DeckGL plugin migration separately? The DeckGL one seems like a more difficult task, but if you're up for it, we can leave them combined :) People will probably want to know more details about how that effort will be tackled.
I'll open the [DISCUSS] thread now, and we can start fielding such questions :)
Comment From: rusackas
I should also say that unless we have feature parity (I'm not sure about that, based on the initial proposal), this would be considered a "breaking change" and would have to be held until a major version of Superset. If it's a smooth transition, I don't think it'd need to wait, but I think users would want some form a migration script in either case. @michael-s-molina has been making several of these and may be able to help point you in the right direction for that stage of the effort.
Comment From: birkskyum
I think it's a good idea to split up these tasks where possible.
Comment From: rusackas
Ok... then I think in effect, this SIP should be considered a proposal to merely to replace Mapbox with MapLibre, on the basis of all the pricing/licensing/support/modernization virtues listed above.
We can open a separate SIP for more details about each of the plugins (any revisions to control panels, migrating to typescript, talk of drilling, etc etc etc) after this general product migration is settled.
I've slightly updated the original description to address this point.
Comment From: rusackas
@birkskyum do you have any idea when you might be able to jump in for the implementation part? I can put this SIP up for a VOTE now if you think it's ready.
Comment From: birkskyum
I can start already next week, and i think the SIP looks ready.
Comment From: JavierMorales2
When could the implementation for the migration from Mapbox to MapLibre be ready?
Comment From: birkskyum
@JavierMorales2 , I haven't deep dived the superset code yet, and I don't know how much different the new Data API framework is, but my guess, based on migrating plotly.js (python graphic tool) is that an initial implementation of a new plugin likely will take 2-3 weeks before it's functional, and that the following QA and release planning and preparation (docs, migration guides etc.) can require additional 1-3 months.
Comment From: rusackas
Vote approved! Let's do the thing!!! :D @birkskyum let us know if you need any help along the way. I'll close the issue as it's approved, but continue to track it through the rest of the SIP kanban board.
Comment From: birkskyum
@rusackas
I'm not sure about how to use the Kanban board - should I write here, or make new tickets in this repo with a [130] tag, or what's the preferred flow so I align?
First steps I've had a good look at how to cut this work into smaller steps which could land independently, to approach the migration in a careful manner. What caught my eye is that react-map-gl, which is used in the legacy deckgl/mapbox plugins is v6.x (which is mapbox-only). Interesting, react-map-gl v7 (it was basically a rewrite) has support for both mapbox and maplibre, so upgrading that first would be a great preparation step before moving to maplibre - and it would potentially allow for publishing new releases of the legacy deckgl and mapbox plugins (same mapbox version) to secure that progress.
Unfortunately, I'm currently a bit stalled keeping my dev-environment up - this docker-compose issue (i think it's just that) seems to be affecting me: - https://github.com/apache/superset/issues/30206
Comment From: victorfonseca
@rusackas
I'm not sure about how to use the Kanban board - should I write here, or make new tickets in this repo with a [130] tag, or what's the preferred flow so I align?
First steps I've had a good look at how to cut this work into smaller steps which could land independently, to approach the migration in a careful manner. What caught my eye is that react-map-gl, which is used in the legacy deckgl/mapbox plugins is v6.x (which is mapbox-only). Interesting, react-map-gl v7 (it was basically a rewrite) has support for both mapbox and maplibre, so upgrading that first would be a great preparation step before moving to maplibre - and it would potentially allow for publishing new releases of the legacy deckgl and mapbox plugins (same mapbox version) to secure that progress.
Unfortunately, I'm currently a bit stalled keeping my dev-environment up - this docker-compose issue (i think it's just that) seems to be affecting me:
Anyone on this?
Comment From: rusackas
@birkskyum sorry for the radio silence. Ping me/us on slack if you ever get stalled here.
I don't think you need to add/track tickets for the work... feel free to open incremental PRs as you see fit.
as for the dev environment, I think you simply need to install zstd on your machine and then run Superset.
I also linked an PR where someone is adding an OSM layer to DeckDL... maybe we need to make sure things are a bit "agnostic" wherever possible, enabling choice/config to pick between tile layers (OSM, MapLibre, whatever). The more "pluggable" things are, the better! Then we might even be able to leave MapBox as a configurable/legacy option for those who want it.
Comment From: SupersetOdT
Hello,
I'm returning to this discussion. I installed version 5.0.0 of Superset and I see that Mapbox is still installed by default.
Does anyone know anything about this very exciting project of migrating to Maplibre?
Kind regards.
Comment From: jiangoxford
Someone proposed a MapLibre migration path for Superset's DeckGL plugin.superset-maplibre Has anyone validated this approach?
Comment From: rusackas
Just checking in on the status of this. We'd love to see it through, but if nobody's going to volunteer to carry the torch, we may close it as Discarded.
Comment From: Aalbedon
I just realized I had not properly set up my notifications until recently and so I've missed all these pings!!
Looking over the history of this, would @birkskyum still be up for the development? Unfortunately I'm not a developer (and still very new to contributing to open source software in any meaningful way), but if there are other ways to help out by leading discussions or helping think through and track some of the features, I'm game! (although I'll need some guidance!)
Comment From: birkskyum
This is still very much on the todo-list, it's frankly just been busy due to project growth.
Since the creation of this proposal ~16 months ago, maplibre-gl weekly downloads from npm have approx. tripled, not to mention that plotly.js / plotly.py has swapped mapbox for maplibre, and those pypi stats quickly made maplibre a popular solution in the python space - so it's an increasingly stable solution that apache/superset would be adopting here, and it would be a great fit for the project.
We've also introduced Globe-mode around last "New Year", and it's a feature that allow for some visualizations to really pop in a way that's hard with a 2d map - such as this global population visualization (more info about it).