posts

Videos:

• Designing ETL Pipelines with Structured Streaming and Delta Lake— How to Architect Things Right
• Recipes for Running Spark Streaming Applications in Production
• Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming And Spark Streaming

This is a bit of a personal post on a tech blog. If the title got you here, then I am sorry there is nothing about tech in this piece, but it is a race report on my most recent race in West Orange, New Jersey (NJ) and a bit about me. On January 16th, I took to the hills of Northern NJ in the South Mountain Reservation to run the Frosty Fat Sass: 6-hour, a race to raise funds for the South Mountain Conservancy. This one was interesting because it was considered semi-virtual; meaning, for the competitors, they marked the trail and told us there were four days to get out there and run. Did I mention it had rained the previous 24 hours? So it was less of running on trail and more of running in mud and creeks.

With college coming to a somewhat anticlimactic end, it was odd to not have known the last day I would spend on the Dickinson campus and physically among so many thoughtful and committed faculty members. With downtime before beginning work, I am thinking back to the days within classes where professors spoke of the formative papers that shaped the computer science domain; among those papers are the names of some of the most notable computer scientists such as Dijkstra and Hoare. While nearly every student has heard these names in lore, we often never found ourselves reading their work - rather we learned of the concepts and contributions of it. In an effort to continue my computer science learning, I decided that why not take the time to read their work and better understand what brought us to where we are in the field today. Please take a moment to check out the infographic-thing I put together for On the Criteria To Be Used in Decomposing Systems into Modules by D.L. Parnas.

This post has been updated with a link to a new GIST which now provides a tool by which all states may be visualized

Over the course of a year learning about and working on an open-source project, the Kick Butlers group , comprised of Ben Cross, Will Freeman, and Gregory Edwards, has traversed the ideas of interacting with an open-source community, applying pair-programming and finding purpose in contributing to a project outside the walls of Dickinson. Each of these exercises provided us with a different perspective and process by which we developed civic disposition and civic skills.

Question: The importance of a singular airport within the airport network can be determined by measuring the degree centrality, the ratio of edges in and out, of it.

In the most recent Seminar discussion, the topic of bias in algorithms was raised as both a point of consideration and debate. This question thought-provoking as it begins to make one wonder where the bias lies. Can code be bias? Is the process by which the code is written a biased process? While both of these things can, undoubtedly, be biased, it is, perhaps, out of the scope of a blog post to cover. Rather, it may be beneficial to narrow the field to an analysis of the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) systems as noted in Algorithmic bias detection and mitigation: Best practices and policies to reduce consumer harms

In the most recent Senior Seminar discussion on facial recognition, the subject of bias was discussed in depth - from areas of criminal justice to the findings of Joy Buolamwini and the significant differences in identification rates based on race and gender. The conversation built from this to include the comparison of other techniques such as fingerprinting. In an article by Jason Brownlee, he notes that the facial recognition tools being developed at Facebook are reported to now have achieved a 97.35% successful identification rate on the Labeled Faces in the Wild dataset. This stands out as it begins to near the 98% identification rate stated for fingerprinting.

When considering driverless cars, the discussion can quickly take a turn to something of an ethical black hole where any argument, despite validity, will descend into nothing more than a circular discussion. While this can be a productive exercise in debate, it often neglects the realities of the space that is driverless cars. This space is quite nuanced with different levels of self-driving needing to be considered.

In the recent seminar discussion, we dove into the dense and difficult topic of ethical components of the computer science major and industry. These things included everything from the alteration of search results to the active creation of worms for the greater good of the internet. While these do include many of the topics, there was, perhaps, one that was glanced over significantly. That is the topic of cybersecurity.

I guess this one is better late than never. Here is a blog post for week 2 coming to you weeks after the actual training. This one is this late as I have been driving across the eastern-seaboard going from NC to NJ and on to CT. With each additional step of travel, I find that I am faced with a new challenge - training in a new place and environment. On top of that, on drives over 5 hours, I have been finding that there is a significant impact on training from the perspective of fatigue. Consistently, the first run following travel feels significantly more difficult as compared to similar runs done previously. Why is that? I am not 100% sure, but I suspect it could be a combination of things from being in a sedentary position when driving for a period of time to the difficulty eating and maintaining a reasonable diet when traveling.

This is week one of the blog and the first week back from being sick over the holidays. Being sick over Christmas and New Years wasn’t a fun time and took its toll. As for what I was sick with, it seemed to be a virus: symptoms included a 101+ fever, congestion and a headache. In the training days following, I found each run to be a grueling effort. With even the least amount of effort, my heart rate was driven up to that of something much faster than I would expect when I was healthy at that pace. At times, I struggled to maintain any momentum.

Well, this is the first of many to come. In about 85 days, I will be undertaking an effort which even a year ago didn’t even seem possible. It’s a 100-mile race - specifically the Greenbrier 100.

In the planning process of this blog post, I found that many of the themes stood out, but, for my experience, the concepts of unfamiliarity and democratic principles stood above all others.

In the recent class discussion, a variety of models for open-source software were visited - from the “support-as-a-service” model of RedHat to the criticisms of the model by Peter Levine. While the discussion covered many points about the success of RedHat, there remains a question about what is the history of RedHat?

When working with or on open source projects, there’s is a collective, social handshake that, in theory, maintains trust for all those who are working on the project. This, in hope, should ensure that all who contribute, use and support the project work with the best interest of the collective community and not themself. While this is the hope, there are horror stories such as that of the bitpay/copay wallet which was supposed to be a secure crypto wallet. With a few short emails and messages, a rogue developer was able to ask to maintain a component and inject malicious code to siphon off coins from the wallet as reported by Ryan Whitwam of Extreme Tech. While this attack could and should be investigated further, the aim of this post is to not do that. Rather, a complement question should be asked: who’s at fault?

Established in the recent discussion of Software Process Models, there is an array of models by which a project can be developed. From the simple code-and-fix approach used by many computer science students with a deadline to the more intricate spiral technique which encourages cyclical development. While many of the models were covered in class, due to time constraints, we did not manage to cover the process of test-driven development. The purpose of this blog is to provide an overview of this approach.

When creating a software application, it is necessary to understand the importance of testing before releasing the product to the public. While many undergraduate institutions will touch on the importance of testing, it can be difficult to see it in an applied context. As both a personal exercise and an effort to assist fellow undergraduates in their transition from academic to industry world, I am taking some time to layout common pipeline for testing in the industry.

During the latest discussion, the class provided only a cursory glance at the many different types of software design patterns. In that, we took about 10 minutes to educate ourselves and present on one of those patters. While that was partially informative, it may have been a bit too much of a superficial glance. I am going to use this blog to dive into the more granular parts of the pattern my group covered: Lock.

When considering the building of any software project, there is an array of challenges that should be taken into consideration to build a stable, scalable, fast, cloud-based, agile, and robust program.

To understand the motivation for the creation of GPL3 (General Public License), it is beneficial to take a look back at the computing atmosphere when GPL2 was first released. The year was 1991, Microsoft Windows had only been around for about half a decade, companies such as Amazon were no more than book stores and the thought of profiting off software and open-source tools was in its infancy. Therefore, the ideas of Torvald, Stallman, and GNU in an industry setting were also young.

With such an array of software architectures covered within An Introduction to Software Architecture by David Garlan and Mary Shaw, the senior seminar class length fell short on providing sufficient definitions for all the options. Specifically, it saw this when defining State Transition Systems. With this platform, I am hoping to take time to shed light on the topic.

As a young student in computer science, the only time that I have heard of the term SVN is in the context of colleagues reminiscing of systems before Git and Github. Admittedly, I, embarrassingly, say that at the start of my junior year of college, I was under the impression that Git was created by Github itself. However, you probably knew that as if you’re reading this painfully limited scope blog.

In the most recent class discussion in the comp-491 seminar, we reviewed the tools which are used for the communication of open source communities. From mailing list, IRC chat to Slack, there are a plethora of different tools which are employed to connect these communities. With the consideration of the foundations of open-source, a question stands: can an open-source project rightly employ a propriety tool for communication or does that threaten the nature of open-source?

“When in the Course of human events it becomes necessary for one people to dissolve the political bands…”

In a time where large firms are constantly competing to raise their bottom line and the battle for every ounce of profit, there does not seem to be an argument for them going out of their way to generate non-proprietary projects. However, the reality is quite the contrary, as we discussed within the class; the remarkable nature of the work by GNU and Stallman is that they have championed basic freedoms of a movement. A movement that is the foundation of much of today’s computing architecture.