Best tools for multimedia reporting Mon, 01 Aug 2016 13:37:30 +0000 en-US hourly 1 https://s0.wp.com/i/webclip.png 32 32 239681374 https://geojournalism.org/2015/02/making-maps-with-jeo/ Willie Shubert Wed, 04 Feb 2015 17:55:42 +0000 Apps Design Maps cartography environmental data https://geojournalism.org/?p=1335 <p>The basic component of a map is its layers&#8212;an individually designed dataset&#8212;that are combined together to tell a story. In JEO 1.o and beyond, maps&#160;are built&#160;utilizing a&#160;layers content type which allows editors to more easily reuse the data they mapped. &#160;These layers are organized and configured in combination with a base map that usually consists [&#8230;]</p> <p>The post <a href="https://geojournalism.org/2015/02/making-maps-with-jeo/">Making Maps with JEO</a> appeared first on <a href="https://geojournalism.org">geojournalism.org</a>.</p> <p>The basic component of a map is its layers—an individually designed dataset—that are combined together to tell a story. In JEO 1.o and beyond, maps are built utilizing a layers content type which allows editors to more easily reuse the data they mapped.  These layers are organized and configured in combination with a base map that usually consists of place names, roads, and major natural features.</p> <div class="wp-caption aligncenter" id="attachment_1337" style="width: 570px"><a href="https://geojournalism.org/wp-content/uploads/2015/02/Ekuatorial_Marine_Traffic.png"><img alt="Map of marine traffic in the Java Sea published by geojournalism project Ekuatorial" aria-describedby="caption-attachment-1337" class="wp-image-1337" decoding="async" fetchpriority="high" height="376" src="https://geojournalism.org/wp-content/uploads/2015/02/Ekuatorial_Marine_Traffic.png" width="560"/></a><p class="wp-caption-text" id="caption-attachment-1337">Map of marine traffic in the Java Sea published by geojournalism project Ekuatorial. The basemap focuses on the readers attention on the ocean areas of the map and is used to contexualize data layers showing shipping lanes and marine protected areas.</p></div> <h1 class="summary-item" id="add-a-new-layer"><a href="#add-a-new-layer"> Add a New Layer</a></h1> <p>JEO’s maps are created by stacking one or more custom thematic layers on top of a basemap. A thematic layer is made from a dataset that illustrates and provides context to the map. Layers are displayed as tiles that are hosted on external servers and connected with the JEO template through APIs and URLs.</p> <p>To add more layers, you can use the “Add MapBox layer” and “Add CartoDB layer” buttons if you want to use layers from these servers. Otherwise, use “Add tile layer.” Inside the box that is created after adding a layer you’ll see three fields that need to be filled out:</p> <ul> <li>Title, which is the name of your layer. It will always be used when you need to refer to this layer (in a menu, for example).</li> <li>ID or URL. This is where you should put the URL of the layer, or, in the case of a MapBox layer, your ID.</li> </ul> <p> </p> <div class="et-tabs-container et_sliderfx_fade et_sliderauto_false et_sliderauto_speed_5000 et_slidertype_top_tabs"> <ul class="et-tabs-control"> <li><a href="#"> Add a Tile Layer </a></li> <li><a href="#"> Add a MapBox Layer </a></li> <li><a href="#"> Add a CartoDB Layer </a></li> <li><a href="#"> Add a Layer Legend </a></li> </ul> <!-- .et-tabs-control --> <div class="et-tabs-content"> <div class="et-tabs-content-wrapper"> <div class="et_slidecontent"> <a href="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.56-PM.png"><img alt="Screen Shot 2016-03-16 at 3.17.56 PM" class="alignnone size-medium wp-image-1569" decoding="async" height="276" sizes="(max-width: 300px) 100vw, 300px" src="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.56-PM-300x276.png" srcset="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.56-PM-300x276.png 300w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.56-PM-1024x943.png 1024w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.56-PM.png 1440w" width="300"/></a> </div> <div class="et_slidecontent"> <img alt="Screen Shot 2016-03-16 at 3.18.02 PM" class="alignnone size-medium wp-image-1570" decoding="async" height="95" sizes="(max-width: 300px) 100vw, 300px" src="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.02-PM-300x95.png" srcset="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.02-PM-300x95.png 300w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.02-PM-1024x325.png 1024w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.02-PM.png 1424w" width="300"/> </div> <div class="et_slidecontent"> <a href="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.08-PM.png"><img alt="Screen Shot 2016-03-16 at 3.18.08 PM" class="alignnone size-medium wp-image-1571" decoding="async" height="137" loading="lazy" sizes="auto, (max-width: 300px) 100vw, 300px" src="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.08-PM-300x137.png" srcset="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.08-PM-300x137.png 300w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.08-PM-1024x467.png 1024w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.08-PM.png 1418w" width="300"/></a> </div> <div class="et_slidecontent"> <a href="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.31-PM.png"><img alt="Screen Shot 2016-03-16 at 3.18.31 PM" class="alignnone size-medium wp-image-1572" decoding="async" height="300" loading="lazy" sizes="auto, (max-width: 261px) 100vw, 261px" src="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.31-PM-261x300.png" srcset="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.31-PM-261x300.png 261w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.18.31-PM.png 552w" width="261"/></a> </div> </div> </div> </div> .et-tabs-container <p>Each layer contains a field for adding the legend to describes the dataset and show how it is symbolized. The legend needs to be written in HTML and is flexible to accept a number of custom styles. Please see this the InfoAmazonia, Ekuatorial, or Third Pole github repository for some example of how these are written.</p> <h1 class="summary-item" id="add-a-new-map"><a href="#add-a-new-map">Add a New Map</a></h1> <p>JEO’s maps are created by stacking one or more custom thematic layers on top of a base map. The Maps Content Type is there location where layers are named and organized into Maps. This is done by configuring the “Maps Setup” section of the form. The basic component of a map is its layers—an individually designed dataset—that are combined together to tell a story. These layers are organized and configured in combination with a base map that usually consists of place names, roads, and major natural features.</p> <h1 class="summary-item" id="choose-a-base-layer"><a href="#choose-a-base-layer">Choose a Base Layer</a></h1> <p>The base layer is the layer is bottom most layer in your map; providing a visual foundation for all of the data added on top. There are a variety of already created base maps you can choose from, including:</p> <ul> <li>OpenStreetMap,</li> <li>Mapquest,</li> <li>Mapquest Satellite,</li> <li>Stamen Toner,</li> <li>Stamen Watercolour,</li> <li>Stamen Terrain,</li> <li>Other (used for when you have your own tilelayer URL)</li> <li>None (in case you want to use a base from an already existing layer in your website).</li> </ul> <h1 class="summary-item" id="add-and-organize-thematic-layers"><a href="#add-and-organize-thematic-layers">Add and Organize Thematic Layers</a></h1> <p>The design of the map is closely related to its order of information. One of the most important concepts in cartography is called visual hierarchy. The idea is focused on making the most important information the most visually prominent in the design. Once you’ve identified the most important information you put it on the top and add supporting data underneath.</p> <p>For example a deforestation map will be ordered like this: forest loss, forest concessions, protected areas, forest height. Layers are displayed according to their visual hierarchy. The higher the layer in the list, the higher its visual priority in the stack of layers that have been added to the map</p> <p>Within the Map CT, there is a searchable interface for finding and selecting the layers you want to add to your map. These layers were named during the first step of this tutorial. When you select multiple layers, their display order is from top to bottom.</p> <p><a href="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.28-PM.png"><img alt="Map CT Layer Selector" class="size-medium wp-image-1573 aligncenter" decoding="async" height="300" loading="lazy" sizes="auto, (max-width: 300px) 100vw, 300px" src="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.28-PM-300x300.png" srcset="https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.28-PM-300x300.png 300w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.28-PM-150x150.png 150w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.28-PM-1021x1024.png 1021w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.28-PM-260x260.png 260w, https://geojournalism.org/wp-content/uploads/2015/02/Screen-Shot-2016-03-16-at-3.17.28-PM.png 1458w" width="300"/></a></p> <p>Once you have selected the layers you want to appear in your map, you the select options that control how your audience can interact with your selected layers. The layers can be:</p> <ul> <li>Fixed – use this if you don’t want to allow the viewer to turn off the layer.</li> <li>Switchable – when you want that the viewer to be able to switch the layer on and off.</li> <li>Swapable – when you would like to permit a choice between a set of layers.</li> </ul> <p>You can review the map design before publishing by clicking on the “UPDATE PREVIEW” button.</p> <h1 class="summary-item" id="defining-the-center-point-scale-and-scope-of-the-map"><a href="#defining-the-center-point-scale-and-scope-of-the-map">Defining the Center Point, Scale and Scope of the Map</a></h1> <p>The last step before publishing the map is the “Map Configurations” section. This is located under the preview map window. Here you can set some limits on the where a user can move to within the map (most maps are not global and should be define). There are three ways to hone in on the interaction.</p> <ul> <li>The center and initial zoom: This is the default view for the map when the page loads.</li> <li>The minimum and maximum zoom: This feature allows you to limit the zoom level to a specific range. Sometime data is now very useful at a high zoom level because it is generalized for a regional map. Other times, the scope of data may only cover a single city. If either of these is the case, limiting the zoom levels will help the map load faster.</li> <li>The limits for navigation of the map: Similar to zoom limits, if the map is not global in scale then it is important to put limits on “panning.” Panning is the action of moving the map from side (2-dimensions) to side rather than zooming in and out (3-dimensions).</li> </ul> <p><a href="https://geojournalism.org/wp-content/uploads/2015/02/Captura-de-Tela-2015-02-04-%C3%A0s-09.38.35.png"><img alt="Captura de Tela 2015-02-04 às 09.38.35" class="alignnone size-full wp-image-1347" decoding="async" height="536" loading="lazy" src="https://geojournalism.org/wp-content/uploads/2015/02/Captura-de-Tela-2015-02-04-%C3%A0s-09.38.35.png" width="922"/></a></p> <p>To create these configurations, you can navigate the map and when you are in the position that you desire, click the buttons “Define current as center and zoom” and “Define current as navigation limits”.</p> <p>Using the other setting, it is possible to:</p> <ul> <li>Turn off the possibility of using the mouse or touchpad to change the zoom</li> <li>Turn on a filter, permitting that the visitors choose the start and end date of the posts which they wish to see</li> <li>Turn on a search field on the map</li> </ul> <div class="author-shortcodes"> <div class="author-inner"> <div class="author-image"> <img alt="" src="https://photos1.meetupstatic.com/photos/member/2/7/7/f/member_18490111.jpeg"/> <div class="author-overlay"></div> </div> <!-- .author-image --> <div class="author-info"> <strong>Willie Shubert</strong> is the Senior Project Coordinator for Internews’ Earth Journalism Network. As a coordinator of a global network of environmental journalists, Willie helps make tools that enable people to connect with each other, find material support, and amplify their local stories to global audiences. In his previous position at National Geographic Magazine, he coordinated translation for the magazine’s 32 local language partners. He holds a degree in Geography from Humboldt State University with concentrations in cartography, environmental economics, and Chinese Studies. Outside of work, he devotes his time to the development of a free school dedicated to community building through education and to collaborative mapping and audio projects. </div> <!-- .author-info --> </div> <!-- .author-inner --> </div> .author-shortcodes <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/> <p>The post <a href="https://geojournalism.org/2015/02/making-maps-with-jeo/">Making Maps with JEO</a> appeared first on <a href="https://geojournalism.org">geojournalism.org</a>.</p> 1335 https://geojournalism.org/2014/04/dustduino-arduino-uno-edition/ Matthew Schroyer Mon, 07 Apr 2014 20:18:05 +0000 Crowdsourcing Data Mobile Data collection environmental data Open Hardware Particulate Matter Sensors https://geojournalism.oeco.org.br/?p=1025 <p>Why Focus on Particulate Matter? The World Health Organization (WHO) took a major step by announcing that air pollution is carcinogenic to humans.WHO also announced they are considering particulate matter, a major component of indoor and outdoor air pollution, as carcinogenic to humans as well. International Agency for Research on Cancer announced today: outdoor&#160;#airpollution&#160;is leading [&#8230;]</p> <p>The post <a href="https://geojournalism.org/2014/04/dustduino-arduino-uno-edition/">DIY air pollution monitoring with DustDuino: Arduino Uno Edition</a> appeared first on <a href="https://geojournalism.org">geojournalism.org</a>.</p> <h1 class="summary-item" id="why-focus-on-particulate-matter"><a href="#why-focus-on-particulate-matter">Why Focus on Particulate Matter?</a></h1> <p>The World Health Organization (WHO) took a major step by announcing that air pollution is carcinogenic to humans.WHO also announced they are considering particulate matter, a major component of indoor and outdoor air pollution, as carcinogenic to humans as well.</p> <p style="padding-left: 60px;">International Agency for Research on Cancer announced today: outdoor <a href="https://twitter.com/search?q=%23airpollution&amp;src=hash">#airpollution</a> is leading environmental cause of <a href="https://twitter.com/search?q=%23cancer&amp;src=hash">#cancer</a> deaths</p> <p style="padding-left: 60px;">— WHO (@WHO) <a href="https://twitter.com/WHO/statuses/390771647461859328">October 17, 2013</a></p> <p>The <a href="https://www.iarc.fr/">International Agency for Research on Cancer</a> (IARC), a special unit inside WHO tasked with promoting international collaborations on cancer research, reached that conclusion after reviewing more than 1000 scientific papers on the carcinogenicity of air pollutants.</p> <p>Air pollution and particulate matter will be included in IARC’s Monograph, which is an encyclopedia of known carcinogens. Particulate matter will be classified as a Group 1 carcinogen, along with tobacco smoke and asbestos. In an IARC press release, the Deputy Head of the Monographs Programme, Dr. Dana Loomis, said that the group’s goal was to “evaluate the air everyone breathes rather than focus on specific air pollutants.”</p> <p>“The results from the reviewed studies point in the same direction: the risk of developing lung cancer is significantly increased in people exposed to air pollution,” he said. This finding elevates the urgency to clean up the air, both outdoors and indoors. But how does one find out the condition of the air in the first place? How healthy is your air?</p> <p><a name="more"></a>Finding that answer can be difficult and expensive. Most Environmental Protection Agency-quality particulate matter monitors send air through a filter, which must be precisely weighed, and produce only one reading a day. Other EPA monitors that sample continuously cost more than <a href="https://wireless.ece.drexel.edu/research/sd_air_quality.pdf">$14,000 each to deploy</a>.</p> <p>Mass concentration is the norm for quantifying airborne particulate matter worldwide, and is usually expressed in micrograms of particulates per cubic meter of air (μg/m3). However, there are an increasing number of affordable particulate sensors coming on the market which rely instead on lasers and LEDs to count the number of particles instead.</p> <p>For <a href="https://www.mentalmunition.com/2013/09/understanding-air-pollution-with-simple.html">more background</a> on the specifics of particulate matter sensing, <a href="https://www.mentalmunition.com/2013/05/dustduino-plan-to-crowdsource.html">please refer</a> to earlier work on Mental Munition regarding particle counting sensors. This post will focus on hacking together your own <a href="https://www.mentalmunition.com/2013/05/dustduino-plan-to-crowdsource.html">DustDuino</a> prototype, which will allow you to monitor dust levels in your home (or back yard) from anywhere you have internet access, for about $100.</p> <p>What will the end product look like, online? <a href="https://xively.com/feeds/499600824">It will look a little like this (my own live DustDuino feed).</a></p> <h1 class="summary-item" id="required-materials"><a href="#required-materials">Required Materials</a></h1> <p>To make a DustDuino, you will need:</p> <div><a href="https://2.bp.blogspot.com/-4VxDLuDJfP8/UmCoqUufsLI/AAAAAAAABzE/03lt24VK5H4/s1600/DustDuino+guts.jpg"><img alt="" border="0" decoding="async" height="215" loading="lazy" src="https://2.bp.blogspot.com/-4VxDLuDJfP8/UmCoqUufsLI/AAAAAAAABzE/03lt24VK5H4/s640/DustDuino+guts.jpg" style="border: 0px;" width="384"/></a></div> <p>1) A WiFi network in your home<br/> 2) <a href="https://www.sparkfun.com/products/11021">Arduino Uno development board</a><br/> 3) <a href="https://www.mouser.com/ProductDetail/Arduino/A000064/?qs=Cvel0QnFNSJK4W8LtutgWg%3D%3D&amp;gclid=CLvD8cGAn7oCFeYWMgodYEEAtA">Arduino Wireless Proto Shield</a><br/> 4) <a href="https://www.seeedstudio.com/depot/grove-dust-sensor-p-1050.html">Shinyei PPD-42 Dust Sensor </a><br/> 5) <a href="https://www.sparkfun.com/products/10822">Sparkfun Roving Networks RN-XV WiFi module</a><br/> 6) <a href="https://www.seeedstudio.com/depot/typeb-usb-cable-for-arduino-diecimila-and-freeduino-p-130.html?cPath=98_100">USB type B cable</a> (for programming the Arduino)<br/> 7) <a href="https://www.seeedstudio.com/depot/wall-adapter-power-supply-9vdc-1a-p-1509.html?cPath=1_4">9V DC power supply</a> (or a 5V USB cell phone charger along with your USB-B cable)<br/> 8) <a href="https://www.sparkfun.com/products/11367">22AWG solid core hookup wire</a> (M/M jumper wires can be used in a pinch)<br/> 9) Soldering equipment (soldering iron, solder, and soldering paste)<br/> 10) Arduino sketch from the <a href="https://github.com/NodeJournalism/DustDuino">DustDuino GitHub repository</a> (<a href="https://github.com/NodeJournalism/DustDuino/archive/master.zip">zipped repo</a> also contains background literature on the PPD-42, mass concentration algorithms, and the DustDuino logo).</p> <h2 class="summary-item" id="step-1-program-the-wifi-module"><a href="#step-1-program-the-wifi-module">Step 1: Program the WiFi module</a></h2> <div><a href="https://2.bp.blogspot.com/-vOlrwmM9lwE/UQ6lUWp4pAI/AAAAAAAAAvQ/4WUGJ2uioMM/s1600/RNXV_PIC.jpg"><img alt="" border="0" decoding="async" height="215" loading="lazy" src="https://2.bp.blogspot.com/-vOlrwmM9lwE/UQ6lUWp4pAI/AAAAAAAAAvQ/4WUGJ2uioMM/s640/RNXV_PIC.jpg" style="border: 0px;" width="384"/></a></div> <p>For the DustDuino to transmit any data, it needs to be connected to your home WiFi network. <a href="https://www.mentalmunition.com/2013/06/step-by-step-guide-to-programming-rn-xv.html">Consult my post on programming the RN-XV for details</a>.</p> <p>Like the Arduino Uno, which serves as the DustDuino’s “brain,” the RN-XV will be on constantly. The RN-XV is set up to connect to your home’s WiFi network as soon as power is connected, and it will remain connected so long as there is power.</p> <h2 class="summary-item" id="step-2-hook-up-the-shinyei-ppd-42-particulate-sensor"><a href="#step-2-hook-up-the-shinyei-ppd-42-particulate-sensor">Step 2: Hook up the Shinyei PPD-42 particulate sensor</a></h2> <div><a href="https://4.bp.blogspot.com/-iyoKpH_NNzA/UmCm11WKsHI/AAAAAAAABy0/SAm5evZGfVQ/s1600/PPD42+connector.jpg"><img alt="" border="0" decoding="async" height="215" loading="lazy" src="https://4.bp.blogspot.com/-iyoKpH_NNzA/UmCm11WKsHI/AAAAAAAABy0/SAm5evZGfVQ/s640/PPD42+connector.jpg" style="border: 0px;" width="384"/></a></div> <p>When you order the Shinyei sensor, it will come packaged with a connector to interface with the <a href="https://www.seeedstudio.com/depot/base-shield-v13-p-1378.html?cPath=132_134">Seeedstudio Grove Shield</a>. Instead of purchasing a whole new shield just to attach a single sensor, I recommend simply sticking either jumper wires, or stripped 22AWG solid core wire into the end of the Grove connector.</p> <p>The black wire on the Grove cable corresponds to ground, so that wire should be connected to GND on the Arduino Wireless Shield. The red wire on the Grove cable will be connected to 5V on the Wireless Shield, while the remaining white wire will be plugged into the digital “8” pin on the Shield.</p> <div><a href="https://3.bp.blogspot.com/-V66NnhE8Y1A/UmCngtjIpCI/AAAAAAAABy8/AtYVdVgDVD8/s1600/PPD42+solder+P2.jpg"><img alt="" border="0" decoding="async" height="215" loading="lazy" src="https://3.bp.blogspot.com/-V66NnhE8Y1A/UmCngtjIpCI/AAAAAAAABy8/AtYVdVgDVD8/s640/PPD42+solder+P2.jpg" style="border: 0px;" width="384"/></a></div> <p>There’s another issue with using the included Grove connector: it doesn’t make use of all the sensor’s available channels. The included connector will only make use of the Shinyei’s P1 channel, which is for counting particles between 1 and 10 microns. But the PPD-42 also has a P2 channel, which counts particles between 2.5 and 10 microns.</p> <p>By measuring both P1 and P2 channels, it’s possible for the DustDuino to differentiate between the PM2.5 and PM10, which are the two major ranges of particles recognized by the EPA and similar agencies. The smaller particulates that measure 2.5 microns and under are of special concern, since those have the potential to enter the blood stream and<a href="https://www.epa.gov/air/urbanair/"> cause the most harm</a>.</p> <p>To measure the P2 channel, strip a solid core 22AWG wire and solder the end of the wire to the second pin on the PPD-42. This is the pin located between GND and 5V. It might be a good idea to use a little hot glue to insulate the connection and provide a little strength.</p> <p>Your new P2 channel will be connected to the digital “9” pin on the Wireless Shield.</p> <h2 class="summary-item" id="step-3-set-up-a-xively-account"><a href="#step-3-set-up-a-xively-account">Step 3: Set up a Xively account</a></h2> <div><a href="https://1.bp.blogspot.com/-eYhwnKz5Le0/UmF-2p8u6-I/AAAAAAAABzw/VNvtQPSwIh0/s1600/xively1.jpg"><img alt="" border="0" decoding="async" height="240" loading="lazy" src="https://1.bp.blogspot.com/-eYhwnKz5Le0/UmF-2p8u6-I/AAAAAAAABzw/VNvtQPSwIh0/s640/xively1.jpg" style="border: 0px;" width="384"/></a></div> <p>All that data being produced by the DustDuino will have to go somewhere, and it might as well go somewhere you can easily access. Xively is a service that <a href="https://xively.com/whats_xively/">markets itself</a> as a “public cloud specifically for the internet of things,” which allows you to view your sensor data from home computer, tablet, or smartphone.</p> <p>That service can get quite expensive, but you can <a href="https://xively.com/signup/">sign up for a free developer account</a>, which can store up to 30 channels of data across five devices (a channel is a single stream of data — like temperature, or humidity, or PM2.5 or PM10). With a free Xively account, any data older than 30 days will be deleted.</p> <p>Xively can walk you through the process of <a href="https://xively.com/dev/tutorials/xively/">setting up an account</a>, but there are some specific things you’ll need to do to make sure it’s set up to receive DustDuino data.</p> <p>You will need to create a device with four channels, named:</p> <ul> <li>PM10</li> <li>PM10count</li> <li>PM25</li> <li>PM25count</li> </ul> <p>PM10count and PM25count channels will store the number of PM10 and PM2.5 particles counted by the Shinyei PPD-42 sensor, while PM10 and PM25 channels will store mass concentration as calculated by the DustDuino code.</p> <p>Record the Feed ID for this device, which you’ll need to insert into the Arduino code later. Also save the name of the device, which you’ll also need for the Arduino code.</p> <p>The last thing you’ll need to do here is to create an API key which will give your DustDuino permission to store data on your Xively account. Every time the DustDuino sends data to Xively, it needs send along this unique key. To make a new API key, or find an existing key, go to “settings,” and then click “Master Keys.”</p> <p>Copy and paste this API key somewhere easy to find, as you’ll need to manually insert it into the Arduino code in our next step.</p> <h2 class="summary-item" id="step-4-code-the-arduino"><a href="#step-4-code-the-arduino">Step 4: Code the Arduino</a></h2> <p>The Arduino is fairly capable “out of the box,” but for more complicated tasks like getting the RN-XV to connect to the Xively server, it’s helpful to have blocks of code already written out. Those blocks of code are known as <a href="https://arduino.cc/en/Reference/Libraries">libraries</a>.</p> <p>Fortunately there’s only one library you’ll need for the DustDuino, and that’s the <a href="https://github.com/harlequin-tech/WiFlyHQ">WiFlyHQ library</a>. The Arduino website has instructions on how to <a href="https://arduino.cc/en/Guide/Libraries">install Arduino libraries</a>.</p> <p>After installing the WiFlyHQ library, open up the DustDuino.ino file in the Arduino IDE. Find the #define APIKEYcode and replace the hash signs with your API Key.</p> <p>Find the very next line, which begins #define FEEDID, and replace the hash signs with the Feed ID.</p> <p>Finally, in the very next line, which begins #define USERAGENT, replace the hash signs with the device name.</p> <p>Save the sketch, and with the Wireless Shield detached from the Arduino Uno, upload the sketch to the Arduino. After the sketch is uploaded, you may re-connect the Wireless Shield. Make sure the switch on the Wireless Shield is set to “Micro” before connecting your DustDuino to a power source.</p> <p>You have several options for powering your DustDuino. You can purchase a separate “wall wart” style 9-volt DC adapter for the Arduino, or you can run power through the Arduino Uno’s USB-B port.</p> <p>If you do decide to supply power through the USB-B port, you may do so with a universal cell charging adapter that has a female USB connector. Most of these chargers supply 5 volts, but just to be sure, make sure to read any text on your cell charger. Another easy way to power the Arduino Uno is to keep it plugged in to your computer’s USB port, but you have to keep your computer powered at all times.</p> <h2 class="summary-item" id="step-5-keep-everything-together"><a href="#step-5-keep-everything-together">Step 5: Keep everything together</a></h2> <div><a href="https://2.bp.blogspot.com/-ePbhNS-BMe4/UmCpWWtqESI/AAAAAAAABzQ/Av7xqDICOV0/s1600/DustDuino+unboxed.jpg"><img alt="" border="0" decoding="async" height="358" loading="lazy" src="https://2.bp.blogspot.com/-ePbhNS-BMe4/UmCpWWtqESI/AAAAAAAABzQ/Av7xqDICOV0/s640/DustDuino+unboxed.jpg" width="640"/></a></div> <p>The Shinyei PPD-42 must be placed upright to work correctly. In other words, it must be placed on a vertical surface, with the connector must be pointing down, and the chrome part of the sensor pointing up.</p> <p>This placement is important to drive air into the sensor. Inside the sensor chamber, at the bottom, is a resistor that heats up when power is connected. With this resistor, the sensor creates a warm updraft of air, which is meant to circulate dust inside the chamber.</p> <p>According to an email from Shinyei customer service, the sensor doesn’t need any kind of fan to blow air inside the sensor chamber. But the chamber must be open to the air you wish to sense.</p> <p>The container doesn’t have to be elaborate. One solution is to secure the sensor to the side of a cardboard box with a pushpin (the sensor has a convenient hole at the top of its circuit board for this kind of thing). If you have a cork board or other board that can be stuck with pins, you also can mount your sensor on a wall.</p> <div><a href="https://2.bp.blogspot.com/-xWLes9mtr2I/UmCsyJYz7eI/AAAAAAAABzY/FFZlGrbmGnM/s1600/dustduino+proto+2+on+wall.jpg"><img alt="" border="0" decoding="async" height="640" loading="lazy" src="https://2.bp.blogspot.com/-xWLes9mtr2I/UmCsyJYz7eI/AAAAAAAABzY/FFZlGrbmGnM/s640/dustduino+proto+2+on+wall.jpg" width="480"/></a></div> <p>At the beginning of this post is a picture of a work in progress: an acrylic cube in which to store the DustDuino. The sides of this cube were cut from an 1/8th inch acrylic sheet using a 50-watt CO2 Epilog laser engraver at the <a href="https://cucfablab.org/">CU Community Fab Lab</a>. The design is not yet perfected, but when finished, the pattern will be made available as a PDF for download.</p> <p>Please note that if you intend to deploy this sensor in your backyard, you will need to make a waterproof housing that still allows the sensor to sample air. You must also ensure the power source to your DIY housing is waterproof, as well. Like the indoor acrylic housing, this is a work in progress.</p> <h2 class="summary-item" id="one-last-word-about-this-prototype"><a href="#one-last-word-about-this-prototype">One last word about this prototype</a></h2> <div><a href="https://1.bp.blogspot.com/-Aav6lLpRvGQ/UmF9eGri4XI/AAAAAAAABzo/STmOFct3W-Y/s1600/dustduino+logo.jpg"><img alt="" border="0" decoding="async" height="230" loading="lazy" src="https://1.bp.blogspot.com/-Aav6lLpRvGQ/UmF9eGri4XI/AAAAAAAABzo/STmOFct3W-Y/s640/dustduino+logo.jpg" style="border: 0px;" width="230"/></a></div> <p>The DustDuino is a work in progress. I cannot make any guarantees as to whether your particular project will work. Having said that, this device has worked great for me so far. So given my personal success, and the recent WHO and IARC findings about particulate matter being carcinogenic, I felt it was time to release instructions and code into the wild.</p> <p>If you put together a DustDuino have it working correctly, or have any suggestions, feel free to drop me a line at mschroyer@gmail.com.</p> <p>Also, if this device has proven useful for you, please consider donating to MentalMunition.com. Funds will go to develop ready-to-deploy DustDuino devices.</p> <div class="author-shortcodes"> <div class="author-inner"> <div class="author-image"> <img alt="" src="https://cucfablab.org/sites/cucfablab.org/files/images/MattSchroyer.jpg"/> <div class="author-overlay"></div> </div> <!-- .author-image --> <div class="author-info"> Matthew Schroyer is a drone and data journalist based in Urbana, Illinois. He is developing drone technology and small Unmanned Aerial Vehicles (sUAV) for use in journalistic enterprises. To this end, he has founded the Professional Society of Drone Journalists (PSDJ), located at DroneJournalism.org. </div> <!-- .author-info --> </div> <!-- .author-inner --> </div> .author-shortcodes <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/> <p>The post <a href="https://geojournalism.org/2014/04/dustduino-arduino-uno-edition/">DIY air pollution monitoring with DustDuino: Arduino Uno Edition</a> appeared first on <a href="https://geojournalism.org">geojournalism.org</a>.</p> 1025 https://geojournalism.org/2013/08/basic-environmental-data-sources-for-mining-information/ Gustavo Faleiros Wed, 07 Aug 2013 21:15:11 +0000 Data environmental data satellite imagery https://dev.cardume.art.br/toolkit/?p=141 <p>Here we bring a collection of sources on different geodata. The way this tutorial is organized follows some of the bigger issues concerning the environmental coverage such as climate change, deforestation, water stress among many others Introduction Journalists and data visualizers cannot complain about the lack of sources on environmental issues. Because of years of [&#8230;]</p> <p>The post <a href="https://geojournalism.org/2013/08/basic-environmental-data-sources-for-mining-information/">Basic Environmental Data: sources for mining information</a> appeared first on <a href="https://geojournalism.org">geojournalism.org</a>.</p> <p>Here we bring a collection of sources on different geodata. The way this tutorial is organized follows some of the bigger issues concerning the environmental coverage such as climate change, deforestation, water stress among many others</p> <h2 class="summary-item" id="introduction"><a href="#introduction">Introduction</a></h2> <p>Journalists and data visualizers cannot complain about the lack of sources on environmental issues. Because of years of scientific research and decades of earth observation, datasets are available in huge amounts and  most often in open formats.</p> <p>Here we bring a collection of sources on different geodata. The way this tutorial is organized follows some of the bigger issues concerning the environmental coverage such as climate change, deforestation, water stress among many others. We also indicate the tutorials you should follow if you would like to manipulate, map and visualize these data.</p> <h2 class="summary-item" id="daily-active-fire"><a href="#daily-active-fire">Daily active fire</a></h2> <p>Fire pixels or hot spots all over the world are monitored by NASA satellites. On the NASA´s <a href="https://earthdata.nasa.gov/" target="_blank">Earth Data</a> website you will find the link for the FIRMS project (Fire Information for Resource Management System). There you will find links for <a href="https://geojournalism.org/glossary/79/" target="_blank" title="CSV">CSV</a>, <a href="https://geojournalism.org/glossary/77/" target="_blank" title="SHP (Shapefile)">Shapefiles (SHP)</a>, <a href="https://geojournalism.org/glossary/78/" target="_blank" title="KML/KMZ">KML</a> (Google Earth format) and <a href="https://geojournalism.org/glossary/721/" target="_blank" title="Web service">web services</a>, covering the last 24hs, 48hs and 72 hours.</p> <p>A good form to keep track of the forest fires in your area of interest is to sign up for email alerts</p> <div class="wp-caption aligncenter" id="attachment_154" style="width: 554px"><a href="https://dev.cardume.art.br/toolkit/wp-content/uploads/2013/08/email-alerts-firms.jpg"><img alt="email alerts firms" aria-describedby="caption-attachment-154" class="wp-image-154" decoding="async" height="326" loading="lazy" src="https://dev.cardume.art.br/toolkit/wp-content/uploads/2013/08/email-alerts-firms.jpg" width="544"/></a><p class="wp-caption-text" id="caption-attachment-154">The website and the page where you can sign up for email alerts on forest fire</p></div> <p style="text-align: left;">For larger datasets, covering several days months and even years of fire pixel, you can use the download tool also available at the website.</p> <p>This is an example of what you can do with these datasets when you map the data. There is information both from fire pixel over the month of July in South America, but also a analysis of the biggest fire events since 2000.<br/> <iframe frameborder="0" height="400" loading="lazy" src="https://infoamazonia.org/embed/?map_only=1&amp;map_id=3502&amp;width=600&amp;height=400&amp;lat=-6.140554782450295&amp;lon=-56.33789062499999&amp;zoom=4" width="600"></iframe></p> <h2 class="summary-item" id="protected-areas"><a href="#protected-areas">Protected areas</a></h2> <p>Limits of National Parks, reserves and indigenous land are essential when creating a map or to contextualize your environmental story. The best source for it is the website <a href="https://www.protectedplanet.net/" target="_blank">Protected Planet</a>, done by the World Commission on Protected Areas, a UN body.</p> <p>There you can download <a href="https://geojournalism.org/glossary/79/" target="_blank" title="CSV">CSV</a>, <a href="https://geojournalism.org/glossary/77/" target="_blank" title="SHP (Shapefile)">SHP</a> and <a href="https://geojournalism.org/glossary/78/" target="_blank" title="KML/KMZ">KML</a> with the boundaries of virtually every protected area in the planet. There is also information on the species existing on these areas and the status of their conservation, either if there are threatened or not of extinction.</p> <p>This is the kind of map you can do by uploading a KML file on Google Maps for example</p> <p style="text-align: center;"><iframe frameborder="0" height="350" loading="lazy" marginheight="0" marginwidth="0" scrolling="no" src="https://maps.google.com/maps/ms?msa=0&amp;msid=217417509911840200241.0004d03ac1c596975f522&amp;hl=en&amp;ie=UTF8&amp;t=h&amp;ll=-7.482345,-55.745025&amp;spn=2.26417,1.01857&amp;output=embed" width="425"></iframe><br/> <small><br/> </small></p> <h2 class="summary-item" id="data-on-the-state-of-the-artic-and-the-antartic"><a href="#data-on-the-state-of-the-artic-and-the-antartic">Data on the state of the Artic and the Antartic</a></h2> <p>Due to Climate Change, a lot of attention has been paid on the state of planets ice environment, or the criosphere.</p> <p><a href="https://nsidc.org/" target="_blank">The</a><a href="https://nsidc.org/" target="_blank"> National</a><a href="https://nsidc.org/" target="_blank"> Snow and Ice Data Center (NSIDC)</a> provides spreadsheets, charts, KMLs and shapefiles that allows easy visualizations and data analysis of the extent of sea ice in the Artic or ice sheets in the Antartic.</p> <p>The downloads are well explained on the <a href="https://nsidc.org/data/docs/noaa/g02135_seaice_index/#daily_data_files" target="_blank">Sea Ice Index Page</a></p> <p>A good way of using this data is to create Google Earth Tours like the one that follows:</p> <p style="text-align: center;"><iframe allowfullscreen="allowfullscreen" frameborder="0" height="360" loading="lazy" src="//www.youtube.com/embed/waMJP7n3_-I" width="640"></iframe></p> <h2 class="summary-item" id="ghg-emissions-on-country-level"><a href="#ghg-emissions-on-country-level">GHG emissions on country level</a></h2> <p>The United Nations Framework Convention On Climate Change (UNFCCC) receives official data from governments on emissions of Greenhouse Gases.</p> <p>The data can be found their website on a section called <a href="https://unfccc.int/ghg_data/items/3800.php" target="_blank">GHG Data</a>. There is data for specific pollutants, such as</p> <p>CO2 – Carbon dioxide<br/> CH4 – Methane<br/> N2O – Nitrous oxide<br/> PFCs – Perfluorocarbons<br/> HFCs – Hydrofluorocarbons<br/> SF6 – Sulphur hexafluoride</p> <p>as well as for the indirect greenhouse gases such as SO2, NOx, CO and NMVOC.</p> <p>The data is provided on XLXs with values divided by country signatories of the Convention. The format allows you to create simple charts for ranking the main emitters, like this.</p> <p><img alt="" class="aligncenter" decoding="async" src="https://docs.google.com/spreadsheet/oimg?key=0Anx0wudPaqDTdFVtMTRPdGxYdzA5MHYxWEVPUHZ2Q3c&amp;oid=6&amp;zx=i68v0ol28snd" width="600"/></p> <h2 class="summary-item" id="forestry-and-deforestation"><a href="#forestry-and-deforestation">Forestry and deforestation</a></h2> <p>Some of the best data compiled on forestry and forest resources can be found on United Nation´s Food and Agriculture Organization (FAO). Every five years they publish their <a href="https://www.fao.org/forestry/fra/fra2010/en/" target="_blank">“Global Forest Resource Assessment”</a> and also they keep and update stats page (<a href="https://faostat3.fao.org/home/index.html#DOWNLOAD" target="_blank">FAOSTAT</a>) on trade and production of forest products, allowing the user to customize their tables for download.</p> <p>On the example bellow is a simple thematic map done with Google Spreadsheet using the numbers of the 2010 global assessment. Here we are showing the countries colored by the size of forest land.</p> <p><img alt="" class="aligncenter" decoding="async" src="https://docs.google.com/spreadsheet/oimg?key=0Anx0wudPaqDTdHE4ZUFVdE9ub3pEMnBjY0Zub1hSTXc&amp;oid=2&amp;zx=y8jieryfiyc4"/></p> <h2 class="summary-item" id="oceans"><a href="#oceans">Oceans</a></h2> <p>The US National Ocean and Atmosphere Administration (NOAA) holds a rich database with information the salinity, oxygen saturation, phosphate concentration, among others indicators of sea health. One visiting the webpage of the <a href="https://www.nodc.noaa.gov/" target="_blank">Oceanographic Data Center (NODC)</a> might get overwhelmed by the amount and complexity of information. The formats are not the usual spreadsheets, but there are some analysis ready to be used.</p> <p>For example, if you access the figures database is possible to find useful information on oceans temperature, like the one below.</p> <p><img alt="" class="aligncenter" decoding="async" src="https://data.nodc.noaa.gov/woa/WOA09F/temperature/JPEG/t_0_0_1.jpg" width="680"/></p> <div class="author-shortcodes"> <div class="author-inner"> <div class="author-image"> <img alt="" src="https://geojournalism.org/wp-content/uploads/2013/09/gustavo-faleiros.jpg"/> <div class="author-overlay"></div> </div> <!-- .author-image --> <div class="author-info"> Knight International Journalism Fellow <strong>Gustavo Faleiros</strong> is an environmental journalist and media trainer who specializes in data-driven journalism. During his first Knight International Journalism Fellowship last year, Faleiros launched <a href="https://infoamazonia.org/" target="_blank">InfoAmazonia</a>, a digital map that uses satellite and other publicly available data to monitor the Amazon rainforest. Before his fellowship, Faleiros helped create the Amazon Communications Network, which trained journalists and produced 200 stories about environmental issues in the region. The network sprang from his earlier accomplishments at the Brazilian environmental news site <a href="https://www.oeco.com.br/" target="_blank">O Eco</a>, where Faleiros introduced the use of satellite images and interactive maps. Faleiros earned a bachelor’s degree in journalism from <a href="https://www.pucsp.br/" target="_blank">Pontificia Universidade Católica do Sao Paulo</a> and a master’s degree in environment, politics and globalization from <a href="https://www.kcl.ac.uk/index.aspx" target="_blank">King’s College London</a>. </div> <!-- .author-info --> </div> <!-- .author-inner --> </div> .author-shortcodes <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/> <p>The post <a href="https://geojournalism.org/2013/08/basic-environmental-data-sources-for-mining-information/">Basic Environmental Data: sources for mining information</a> appeared first on <a href="https://geojournalism.org">geojournalism.org</a>.</p> 141 https://geojournalism.org/2013/07/cleaning-data-with-refine/ Michael Bauer Wed, 17 Jul 2013 22:36:29 +0000 Data data wrangling environmental data https://dev.cardume.art.br/toolkit/?p=94 <p>Open Refine (previously Google Refine) is a data cleaning software that uses your web browser as an interface. This means it will look like it runs on the internet but all your data remains on your machine and you do not need internet connection to work with it. The main aim of Refine is to [&#8230;]</p> <p>The post <a href="https://geojournalism.org/2013/07/cleaning-data-with-refine/">Cleaning Data with Refine</a> appeared first on <a href="https://geojournalism.org">geojournalism.org</a>.</p> <p>Open Refine (previously Google Refine) is a data cleaning software that uses your web browser as an interface. This means it will look like it runs on the internet but all your data remains on your machine and you do not need internet connection to work with it.</p> <p><span id="more-94"></span></p> <p>The main aim of Refine is to help you exploring and cleaning your data before you use it further. It is built for large datasets – so don’t worry as long as your spreadsheets can hold the information, refine can as well.</p> <h2 class="summary-item" id="creating-a-new-project"><a href="#creating-a-new-project">Creating a new Project</a></h2> <p>To work with your data in Refine you need to start a new project.</p> <p><strong>Walkthrough: Creating a Refine project</strong><br/> </p><div class="et-tabs-container et_sliderfx_fade et_sliderauto_false et_sliderauto_speed_5000 et_slidertype_top_tabs"> <ul class="et-tabs-control"> <li><a href="#"> Step 1 </a></li> <li><a href="#"> Step 2 </a></li> <li><a href="#"> Step 3 </a></li> </ul> <!-- .et-tabs-control --> <div class="et-tabs-content"> <div class="et-tabs-content-wrapper"> <div class="et_slidecontent"> <ol> <li>Start Refine – this will open a browser window pointing to <a href="https://127.0.0.1:3333" target="_blank">https://127.0.0.1:3333</a> if this doesn’t happen open the link with your browser directly</li> <li>Create a new project: On the left tab select the “Create Project” tab:<img alt="Cleaning Data with Refine" decoding="async" height="183" loading="lazy" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia01.png" style="margin: 15px 610px 0px 0px;" width="170"/></li> </ol> </div> <div class="et_slidecontent"> <ol> <li>Click on “Choose Files” to choose your downloaded file and click on “next” – you can also use the URL to the <a href="https://geojournalism.org/glossary/79/" target="_blank" title="CSV">CSV</a> directly if your data is hosted on the web.</li> <li>You will get a preview on how refine will interpret your data – if you have selected a well formatted CSV or other file: this should be pretty automatic.</li> <li>Review the preview carefully to make sure the data looks right. Double check character encoding. Much, but not all data uses UTF-8 these days, but make sure you don’t see any funny characters in preview.</li> <li>In our Case all the Column have no column headings – since the first row of the file is actually empty.<img alt="Cleaning Data with Refine" class="size-full wp-image-278 aligncenter" decoding="async" height="255" loading="lazy" sizes="auto, (max-width: 531px) 100vw, 531px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia03.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia03.png 531w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia03-300x144.png 300w" style="margin-top: 15px;" width="531"/></li> </ol> </div> <div class="et_slidecontent"> <ol> <li>If you check further down, you’ll notice that the column headings for the data are in rows 5 and 6.</li> <li>Let’s tell refine where to find column information: On the bottom you can find settings on how refine should look for column information.</li> <li>We’ll tell it to ignore the first 5 lines, take the next 2 lines as column labels and ignore one further row:</li> </ol> <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="size-full wp-image-279 aligncenter" decoding="async" height="99" loading="lazy" sizes="auto, (max-width: 485px) 100vw, 485px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia04.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia04.png 485w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia04-300x61.png 300w" style="margin-top: 15px;" width="485"/></p> <ol> <li>Great – now the preview looks much nicer.</li> <li>Name your project in the box on the top right side and click on “Create Project”.</li> <li>The project will open in the project view, this is the basic interface you are going to work with: by default refine shows only 10 rows of data, you can change this on the bar above the data rows. Also you can use the navigation on the right to see the next or previous rows.</li> </ol> </div> </div> </div> </div> .et-tabs-container <p></p> <p>You now have successfully created your first refine project. Remember: although it runs in a web-browser, the Refine server is still on your machine – all the data is there (so no worries if you handle sensitive information)</p> <h2 class="summary-item" id="sorting-and-facetting"><a href="#sorting-and-facetting">Sorting and Facetting</a></h2> <p>Once we created our project, let’s go and explore the data and the Refine interface a bit. Using Refine might be intimidating at first, since it seems so different from spreadsheets, once you get used to it you will notice how easily you can do things with it.</p> <p>One of the commonly used functions in spreadsheets is sorting and filtering data – to figure out minima, maxima or things about certain categories. Refine can do the same thing.</p> <p><strong>Walkthrough: Sorting rows</strong><br/> </p><div class="et-tabs-container et_sliderfx_fade et_sliderauto_false et_sliderauto_speed_5000 et_slidertype_top_tabs"> <ul class="et-tabs-control"> <li><a href="#"> Step 1 </a></li> <li><a href="#"> Step 2 </a></li> <li><a href="#"> Step 3 </a></li> </ul> <!-- .et-tabs-control --> <div class="et-tabs-content"> <div class="et-tabs-content-wrapper"> <div class="et_slidecontent"> <ol> <ol> <li>Refines handles data similar to a spreadsheet: you have rows, columns and cells – a cell is a field defined by a row and a column.</li> <li>To sort your rows based on a specific column click on the small downward triangle next to the column.<img alt="Cleaning Data with Refine" decoding="async" height="300" loading="lazy" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia05-154x300.png" style="margin: 15px 610px 0 0;" width="154"/></li> </ol> </ol> </div> <div class="et_slidecontent"> <ol> <li>Select “Sort…” to open the sorting dialog.</li> <li>You can select what to sort the values as and then what order to sort in. (We’ll sort in text, since for now we only have text columns). <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-281" decoding="async" height="359" loading="lazy" sizes="auto, (max-width: 424px) 100vw, 424px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia06.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia06.png 424w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia06-300x254.png 300w" style="margin-top: 15px;" width="424"/></p> </li> </ol> </div> <div class="et_slidecontent"> <ol> <li>Click “OK” and your rows will be sorted based on the column.</li> <li>To undo the sort, click on the column options again, select “sort” then “remove sort”</li> </ol> <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-282" decoding="async" height="319" loading="lazy" sizes="auto, (max-width: 410px) 100vw, 410px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia07.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia07.png 410w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia07-300x233.png 300w" style="margin-top: 15px;" width="410"/></p> </div> </div> </div> </div> .et-tabs-container <p></p> <p>The other frequently used function in Spreadsheets is filtering – in Refine this is called facetting. Facetting in Refine is really powerful – you will see in most of the rest of the Recipe we’ll use facets.</p> <p><strong>Walkthrough: Facetting rows based on a column</strong><br/> </p><div class="et-tabs-container et_sliderfx_fade et_sliderauto_false et_sliderauto_speed_5000 et_slidertype_top_tabs"> <ul class="et-tabs-control"> <li><a href="#"> Step 1 </a></li> <li><a href="#"> Step 2 </a></li> <li><a href="#"> Step 3 </a></li> <li><a href="#"> Step 4 </a></li> </ul> <!-- .et-tabs-control --> <div class="et-tabs-content"> <div class="et-tabs-content-wrapper"> <div class="et_slidecontent"> <ol> <li>Select the column options for the column you want to facet with</li> <li>Select “Facet”<img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-283" decoding="async" height="300" loading="lazy" sizes="auto, (max-width: 333px) 100vw, 333px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia08.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia08.png 333w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia08-300x270.png 300w" style="margin-top: 15px;" width="333"/></li> </ol> </div> <div class="et_slidecontent"> <ol> <li>You can facet differently for text, numbers or dates – let’s facet as text – click on “Text facet”</li> <li>This will open a facet in the left bar<img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-284" decoding="async" height="385" loading="lazy" sizes="auto, (max-width: 301px) 100vw, 301px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia09.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia09.png 301w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia09-234x300.png 234w" style="margin-top: 15px;" width="301"/></li> </ol> </div> <div class="et_slidecontent"> <ol> <li>Now select one or more of the choices and you’ll see how your data rows are limited to just those selected.</li> <li>Of course you can add more than one facet and thus filter more than once.</li> <li>Let’s add another facet: Note that one of the columns contain dates: we can add a timeline facet to let us select a date range <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-285" decoding="async" height="300" loading="lazy" sizes="auto, (max-width: 342px) 100vw, 342px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia10.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia10.png 342w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia10-300x263.png 300w" style="margin-top: 15px;" width="342"/></p> </li> </ol> </div> <div class="et_slidecontent"> <ol> <li>This will open a new facet looking like: <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-286" decoding="async" height="173" loading="lazy" sizes="auto, (max-width: 302px) 100vw, 302px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia11.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia11.png 302w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia11-300x171.png 300w" style="margin-top: 15px;" width="302"/></p> </li> </ol> <ol> <li>You can select a time frame by moving the indicators left and right.</li> </ol> </div> <div class="et_slidecontent"> </div> </div> </div> </div> .et-tabs-container <p></p> <h2 class="summary-item" id="reconciling-columns"><a href="#reconciling-columns">Reconciling Columns</a></h2> <p>Sometimes humans make mistakes when they enter data – they mistype city names or put in characters they can not see but the computer can. (For example, you can add a simple space at the end of a name and the computer will think they are different). For this let’s create a text facet for the cities:</p> <p><strong>Walkthrough: Reconciling Columns</strong><br/> </p><div class="et-tabs-container et_sliderfx_fade et_sliderauto_false et_sliderauto_speed_5000 et_slidertype_top_tabs"> <ul class="et-tabs-control"> <li><a href="#"> Step 1 </a></li> <li><a href="#"> Step 2 </a></li> </ul> <!-- .et-tabs-control --> <div class="et-tabs-content"> <div class="et-tabs-content-wrapper"> <div class="et_slidecontent"> <ol> <li>Create a text facet for the City column</li> <li>Scroll down where it says La Paz: see how many different ways there are to write la paz?<img alt="Cleaning Data with Refine" decoding="async" height="181" loading="lazy" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia12.png" style="margin-top: 15px;" width="297"/></li> </ol> </div> <div class="et_slidecontent"> <ol> <li>Let’s fix this! This is so common that refine has a built in mechanism for detecting these kind of differences.</li> <li>Click on the “Cluster” button on the top right in the facet. <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter wp-image-288" decoding="async" height="367" loading="lazy" sizes="auto, (max-width: 591px) 100vw, 591px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia13.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia13.png 739w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia13-300x186.png 300w" style="margin-top: 15px;" width="591"/></p> </li> </ol> <ol> <li>Now check all the things Refine correctly noticed as similar</li> <li>Click on the “Merge Selected &amp; Re-Cluster” button. Play with the keying function and methods on top to identify further possible merges.</li> </ol> </div> </div> </div> </div> .et-tabs-container <p></p> <h2 class="summary-item" id="making-city-names-look-nice"><a href="#making-city-names-look-nice">Making city names look nice</a></h2> <p>Did you notice how most of the Cities are all uppercase? It’s rare to read them like this. And maybe you want to have nicer looking names: No problem. Refine supports this.</p> <p><strong>Walkthrough: Changing Case in Refine</strong></p> <ol> <li>Let’s change the case in our city column from all uppercase to titlecase</li> <li>To do this, open the column options, go to edit cells -&gt; common transforms -&gt; to titlecase <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-289" decoding="async" height="475" loading="lazy" sizes="auto, (max-width: 516px) 100vw, 516px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia14.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia14.png 516w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia14-300x276.png 300w" style="margin-top: 15px;" width="516"/></p> </li> </ol> <ol> <li>Tada – magically your names have been converted.</li> </ol> <p><strong>Congratulations! </strong>You successfully cleaned up a dataset using Refine!</p> <div class="author-shortcodes"> <div class="author-inner"> <div class="author-image"> <img alt="" src="https://farm9.staticflickr.com/8321/7937480904_6f29810da7.jpg"/> <div class="author-overlay"></div> </div> <!-- .author-image --> <div class="author-info"> <strong>Michael</strong> lives in Vienna and works as a Data Wrangler with the Open Knowledge Foundation mostly around the School of Data. After a detour in biomedical research, where he learned to love datawrangling, he spent some time doing advocacy for his passion: freedom in the digital age. He joined the Open Knowledge Foundation to satisfy his curiosity. He will gladly jump on any topic that you point him to. – See more at: <a href="https://okfn.org/about/team/#Michael_Bauer_8212_Data_Wrangler_School_of_Data" target="_blank">https://okfn.org/about/team/#Michael_Bauer_8212_Data_Wrangler_School_of_Data</a> </div> <!-- .author-info --> </div> <!-- .author-inner --> </div> .author-shortcodes <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/> <p>The post <a href="https://geojournalism.org/2013/07/cleaning-data-with-refine/">Cleaning Data with Refine</a> appeared first on <a href="https://geojournalism.org">geojournalism.org</a>.</p> 94