Cline

# Testing Methodologies Standards for Crystal This document outlines the testing methodologies standards for Crystal projects. Following these guidelines will help ensure the creation of reliable, maintainable, and performant Crystal applications. ## 1. General Testing Principles * **Do This:** Embrace a test-driven development (TDD) or behavior-driven development (BDD) approach whenever possible. Write tests *before* implementing the corresponding functionality. * **Why:** TDD/BDD forces you to think about the functionality's purpose and interface before implementation, leading to better design and more comprehensive test coverage. * **Don't Do This:** Write tests as an afterthought. Neglecting testing leads to fragile codebases, increased debugging time, and higher maintenance costs. * **Do This:** Aim for high test coverage (ideally above 80%). Use tools like "crystal spec" with coverage reporting to measure coverage. * **Why:** Higher coverage significantly reduces the risk of regressions and undetected bugs. Be mindful that coverage is only part of the equation; test quality also matters. * **Don't Do This:** Equate high coverage with perfect testing. High coverage without meaningful assertions is useless. Focus on testing critical paths and edge cases. * **Do This:** Write focused and independent tests. * **Why:** Independent tests can be run in parallel, reducing test suite execution time. Focused tests make debugging failing tests easier. * **Don't Do This:** Create tests that depend on specific data or external state. Use mocking or test doubles to isolate units under test. * **Do This:** Maintain your test suite. Refactor tests as your codebase evolves. Delete or update outdated tests. * **Why:** An unmaintained test suite becomes a burden rather than an asset. Regularly pruning and updating tests ensures continued relevance and value. ## 2. Unit Testing ### 2.1. Purpose * Unit tests verify the behavior of individual components (classes, modules, or functions) in isolation. The goal is to validate that each unit performs its intended function correctly. ### 2.2. Standards * **Do This:** Use the built-in "spec" framework for unit testing. * **Why:** It's the standard testing library and integrates seamlessly with Crystal. """crystal require "spec" describe "Calculator" do it "adds two numbers correctly" do calculator = Calculator.new calculator.add(2, 3).should eq(5) end it "subtracts two numbers correctly" do calculator = Calculator.new calculator.subtract(5, 2).should eq(3) end end """ * **Do This:** Follow the Arrange-Act-Assert (AAA) pattern in your tests. * **Why:** AAA promotes clarity and readability by dividing the test into distinct phases. """crystal it "calculates area of a rectangle" do # Arrange rectangle = Rectangle.new(width: 5, height: 10) # Act area = rectangle.area # Assert area.should eq(50) end """ * **Do This:** Use mocks and stubs to isolate the unit under test. * **Why:** Mocks and stubs replace dependencies with controlled substitutes, preventing side effects and ensuring predictable test results. """crystal require "spec" require "mock" # Install: shards install mock class ExternalService def get_data # Imagine this performs an expensive network call raise "Not implemented" end end class MyClass def initialize(@service : ExternalService) end def process_data data = @service.get_data "Processed: #{data}" end end describe MyClass do it "processes data from external service" do mock_service = Mock.new(ExternalService) mock_service.should_receive(:get_data).and_return("mocked data") my_class = MyClass.new(mock_service.object) result = my_class.process_data result.should eq("Processed: mocked data") mock_service.verify end end """ * **Don't Do This:** Make your tests too tightly coupled to the implementation details. * **Why:** Tests that directly depend on internal implementation become brittle and break easily when the code is refactored. Favor testing public interfaces and observable behavior. * **Do This:** Test for exceptions or error conditions. * **Why:** Verifying that your code handles errors gracefully is crucial for reliability. """crystal it "raises an exception when dividing by zero" do calculator = Calculator.new expect_raises(ArgumentError) do calculator.divide(10, 0) end end """ * **Do This:** Use data providers or parameterized tests to cover various input values with a single test case. """crystal require "spec" describe "StringUtil" do { "hello world" => "HelloWorld", "foo bar baz" => "FooBarBaz", "a b c" => "ABC" }.each do |input, expected| it "capitalizes words in '#{input}' to '#{expected}'" do StringUtil.capitalize_words(input).should eq(expected) end end end """ ### 2.3 Common Anti-Patterns 1. **Testing private methods directly:** Generally, avoid this unless absolutely necessary. Focus on testing public interfaces. If a private method requires extensive testing, it might indicate that it should be extracted into its own class. 2. **Over-mocking:** Mocking every single dependency can lead to tests that are too specific and don't actually verify meaningful behavior. Use mocks strategically, focusing on external services or complex dependencies. Prefer integration tests for verifying interactions between internal components when appropriate. 3. **Ignoring edge cases:** Make sure to consider edge cases, boundary conditions, and invalid input when writing unit tests. These are often the source of bugs. ## 3. Integration Testing ### 3.1. Purpose * Integration tests verify the interaction between multiple components or systems. They ensure that units work together correctly. This might involve testing components within your application or testing interactions with external services (databases, APIs, etc.). ### 3.2. Standards * **Do This:** Use integration tests to verify that different parts of your application work together as expected. * **Why:** Unit tests isolate components, but integration tests ensure that these components interact correctly in a real-world scenario. """crystal require "spec" require "sqlite3" describe "User Registration" do it "creates a new user in the database" do # Arrange db = SQLite3::Database.new(":memory:") # Use an in-memory database for testing db.execute <<-SQL CREATE TABLE users ( id INTEGER PRIMARY KEY, username VARCHAR(255) ); SQL user_repo = UserRepository.new(db) username = "testuser" # Act user_repo.create(username) # Assert result = db.execute("SELECT username FROM users WHERE username = ?", username) result.should eq([[username]]) db.close end end class UserRepository def initialize(@db : SQLite3::Database) end def create(username : String) @db.execute("INSERT INTO users (username) VALUES (?)", username) end end """ * **Do This:** Use real dependencies whenever possible, but consider using test databases or mock APIs to avoid side effects and ensure reproducible tests. * **Why:** Using real dependencies provides a more realistic testing environment. * **Do This:** Properly isolate integration tests using transactions or other techniques to prevent interference between tests. * **Why:** Shared state between integration tests can lead to flaky and unpredictable results. """crystal # Example using transactions for database integration tests before_each do @db = SQLite3::Database.new(":memory:") @db.execute("BEGIN TRANSACTION") # Start a transaction @user_repo = UserRepository.new(@db) @db.execute <<-SQL CREATE TABLE users ( id INTEGER PRIMARY KEY, username VARCHAR(255) ); SQL end after_each do @db.execute("ROLLBACK TRANSACTION") # Rollback the transaction @db.close end """ Using this pattern ensures each tests starts with a clean database and avoids interfering with other tests. * **Don't Do This:** Run integration tests against production databases or live APIs. * **Why:** Integration tests should not modify or depend on production data. ### 3.3. Common Anti-Patterns 1. **Skipping integration tests:** Relying solely on unit tests can lead to missed integration issues, where individual components work fine but fail to interact correctly. 2. **Overlapping unit and integration tests:** Clearly define the scope of each type of test. Unit tests focus on individual units, while integration tests verify interactions between components. 3. **Ignoring database migrations:** Always test database migrations as part of your integration test suite to ensure that schema changes are applied correctly. ## 4. End-to-End (E2E) Testing ### 4.1. Purpose * E2E tests simulate real user interactions with the application, verifying that the entire system works correctly from start to finish. These tests typically involve a user interface (web browser, mobile app) and multiple backend services. ### 4.2. Standards * **Do This:** Use tools like [Selenium](https://www.selenium.dev/) or [Capybara](https://github.com/teamcapybara/capybara) (via appropriate Crystal bindings if available, or through system calls to external processes) for web application E2E testing. Consider libraries such as playwright-cr for more modern browser automation. * **Why:** These tools allow you to automate browser interactions and verify the behavior of your application from a user's perspective. """crystal # Illustration of a simple E2E test concept (requires external setup and browser driver) it "allows a user to register and log in" do # Simulate user actions: # 1. Open the registration page # 2. Fill in the registration form # 3. Submit the form # 4. Verify that the user is redirected to the login page # 5. Fill in the login form # 6. Submit the form # 7. Verify that the user is logged in and redirected to the dashboard # Assertions: # - Check that the user is logged in by verifying the presence of a welcome message # - Check that the user can access protected resources # - Check that the user can log out successfully end """ Note: The above is a conceptual example since a full selenium example requires significant setup including selenium server, appropriate drivers for your chosen web browser, and potentially system call bindings from crystal. * **Do This:** Write E2E tests that cover critical user flows, such as registration, login, checkout, etc. * **Why:** Focus on the most important user journeys to ensure that the core functionality of your application is working as expected. * **Do This:** Use a dedicated testing environment for E2E tests to avoid impacting production data. * **Why:** E2E tests often involve modifying data, so it's crucial to isolate them from production environments. * **Don't Do This:** Make your E2E tests too granular. Focus on testing entire user flows rather than individual UI elements. * **Why:** E2E tests are more expensive to run and maintain than unit or integration tests, so it's important to strike a balance between coverage and efficiency. ### 4.3. Common Anti-Patterns 1. **Ignoring E2E tests:** Skipping E2E tests can lead to critical bugs in the user interface or core workflows. 2. **Flaky E2E tests:** E2E tests are often prone to flakiness due to timing issues, network latency, or external dependencies. Implement retry mechanisms and improve test isolation to mitigate flakiness. 3. **Slow E2E tests:** Optimize your E2E test suite to reduce execution time. Run tests in parallel and use efficient selectors to locate UI elements. ## 5. Performance Testing ### 5.1. Purpose * Performance testing assesses the responsiveness, stability, and scalability of your application under various load conditions. It helps identify bottlenecks and optimize performance. ### 5.2. Standards * **Do This:** Use tools like [wrk](https://github.com/wg/wrk) or [hey](https://github.com/rakyll/hey) or [vegeta](https://github.com/tsenart/vegeta) to simulate load on your application. * **Why:** These tools allow you to measure the performance of your application under different load scenarios. """bash # Example using wrk to benchmark a web server wrk -t12 -c400 -d30s http://localhost:3000/ # -t: Number of threads # -c: Number of connections # -d: Duration of the test """ * **Do This:** Use profiling tools to identify performance bottlenecks in your code. Crystal's standard library includes "Profiler". * **Why:** Profilers help you pinpoint the areas of your code that are consuming the most resources. """crystal require "profiler" Profiler.start # Your code here Profiler.stop Profiler.print(STDOUT) """ The Profiler provides basic CPU profiling. More advanced tools may integrate with the operating system directly. Be sure to read the documentation for the tool you are using, and understand the output it produces. * **Do This:** Define performance metrics and set performance budgets for your application. * **Why:** Performance budgets help you track and maintain the performance of your application over time. Examples may include: request latency, throughput, error rate, CPU usage, memory usage. * **Do This:** Automate performance tests and integrate them into your CI/CD pipeline. * **Why:** Automated performance testing allows you to detect performance regressions early in the development cycle. * **Don't Do This:** Ignore performance testing until late in the development cycle. * **Why:** Performance issues can be difficult and costly to fix late in the development cycle. ### 5.3. Common Anti-Patterns 1. **Lack of performance testing:** Neglecting performance testing can lead to slow and unresponsive applications. 2. **Testing in unrealistic environments:** Performance tests should be conducted in an environment that closely resembles the production environment. 3. **Ignoring performance regressions:** Monitor performance metrics and address regressions promptly. ## 6. Security Testing ### 6.1. Purpose * Security testing identifies vulnerabilities in your application that could be exploited by attackers. It helps ensure the confidentiality, integrity, and availability of your data. ### 6.2. Standards * **Do This:** Perform static analysis of your code to identify potential security vulnerabilities. Consider using tools that look for common coding flaws, such as those that might lead to injection attacks. * **Why:** Static analysis can detect vulnerabilities early in the development cycle before they are introduced into production. * **Do This:** Perform dynamic analysis to identify vulnerabilities in your application at runtime. * **Why:** Dynamic analysis can detect vulnerabilities that are not detectable through static analysis. * **Do This:** Conduct penetration testing to simulate real-world attacks on your application. * **Why:** Penetration testing helps identify vulnerabilities that might be missed by automated tools. * **Do This:** Follow security best practices, such as input validation, output encoding, and least privilege. * **Why:** These practices help prevent common security vulnerabilities. * **Don't Do This:** Ignore security testing or assume that your application is secure by default. * **Why:** Security vulnerabilities can have serious consequences for your organization and your users. ### 6.3. Common Anti-Patterns 1. **Lack of security testing:** Neglecting security testing can leave your application vulnerable to attack. 2. **Relying solely on automated tools:** Automated tools can be helpful, but they cannot replace human expertise. 3. **Ignoring security updates:** Keep your dependencies up to date to patch known security vulnerabilities. ## 7. Test Doubles: Mocks, Stubs, and Fakes * **Understanding the Purpose:** Test doubles are essential tools for isolating units under test. ### 7.1. Mocks * **Do This:** Use mocks to verify that a dependent object is called with the correct arguments and in the correct order. Mocks are about behavior verification. * **Why:** Mocks allow you to assert that specific interactions occur between the unit under test and its dependencies """crystal require "spec" require "mock" class Notifier def send_email(recipient : String, message : String) puts "Sending email to #{recipient}: #{message}" # In real code, this would send an actual email end end class UserRegistration def initialize(@notifier : Notifier) end def register(email : String) # Registration logic here message = "Welcome to our service!" @notifier.send_email(email, message) end end describe UserRegistration do it "sends a welcome email after registration" do mock_notifier = Mock.new(Notifier) mock_notifier.should_receive(:send_email).with("test@example.com", "Welcome to our service!") registration = UserRegistration.new(mock_notifier.object) registration.register("test@example.com") mock_notifier.verify end end """ ### 7.2. Stubs * **Do This:** Use stubs to provide canned responses to method calls. Stubs are about state verification. * **Why:** Stubs allow you to control the return values of dependencies to simulate different scenarios. """crystal require "spec" class PaymentGateway def process_payment(amount : Float) : Bool raise "Not Implemented" end end class OrderService def initialize(@gateway : PaymentGateway) end def process_order(amount : Float) : Bool @gateway.process_payment(amount) end end describe OrderService do it "successfully processes an order when payment is successful" do # Arrange successful_gateway = Object.new.tap do |gateway| def gateway.process_payment(amount : Float) : Bool true # Stubbed to always return true end end order_service = OrderService.new(successful_gateway) # Act result = order_service.process_order(100.0) # Assert result.should be_true end it "fails to process order when payment fails" do # Arrange failed_gateway = Object.new.tap do |gateway| def gateway.process_payment(amount : Float) : Bool false # Stubbed to always return false end end order_service = OrderService.new(failed_gateway) # Act result = order_service.process_order(100.0) # Assert result.should be_false end end """ ### 7.3. Fakes * **Do This:** Use fakes to create simplified implementations of dependencies that mimic the behavior of the real ones but are easier to control in tests (e.g., in-memory database). * **Why:** Fakes provide a lightweight alternative to real dependencies, improving test performance and isolation. """crystal require "spec" class UserRepository def save(user : User) raise "Not Implemented" end def find(id : Int32) : User? raise "Not Implemented" end end class InMemoryUserRepository < UserRepository def initialize @users = {} of Int32 => User @next_id = 1 end def save(user : User) if user.id.nil? user.id = @next_id @next_id += 1 end @users[user.id] = user end def find(id : Int32) : User? @users[id] end end struct User property id : Int32? property name : String end describe InMemoryUserRepository do it "saves and retrieves a user" do repo = InMemoryUserRepository.new user = User.new(id: nil, name: "John Doe") repo.save(user) retrieved_user = repo.find(user.id) retrieved_user.should eq(user) end end """ ## 8. Version Specific Considerations * **Crystal 1.0+:** Ensure compatibility with the latest stable version of Crystal. Use features introduced in recent versions to write more concise and efficient tests. Pay attention to deprecation warnings and update your tests accordingly. Refer to the Crystal release notes for specifics. * **Shards:** Keep your testing dependencies (like "mock.cr") up to date by using "shards update". * **Crystal Tooling:** Utilize "crystal tool format" and Ameba as part of your pre-commit hooks or CI process to ensure consistent code formatting and identify potential issues. By adhering to these testing methodologies standards, Crystal developers can build robust, maintainable, and high-performing applications. Remember to adapt these guidelines to the specific needs and context of your projects.

# Tooling and Ecosystem Standards for Crystal This document outlines the recommended tooling and ecosystem standards for Crystal development. These guidelines promote consistency, maintainability, and performance across projects, and serve as context for AI coding assistants. ## 1. Dependency Management with Shards Shards is the official dependency manager for Crystal. Using it correctly is crucial for project maintainability. ### 1.1. Using Shards.lock **Do This:** Always commit "shards.lock" to your version control system. **Why:** "shards.lock" ensures that everyone on the team uses the exact same versions of dependencies. Without it, you risk dependency conflicts and inconsistent behavior across environments. **Example:** """ # Correct - shards.lock is present .gitignore shards.yml shards.lock src/ """ **Don't Do This:** Exclude "shards.lock" from your Git repository. ### 1.2. Specifying Dependency Versions **Do This:** Use semantic versioning when specifying dependencies in "shards.yml". Utilize version constraints like "~>", ">=", and "<" to define acceptable version ranges. **Why:** Semantic versioning allows you to receive bug fixes and minor features without introducing breaking changes. Specific version constraints provide control over which versions are allowed, mitigating risks associated with unexpected updates. **Example:** """yaml # shards.yml dependencies: kemal: github: kemalcr/kemal version: "~> 1.5" # Allow versions 1.5.x up to (but not including) 1.6.0 jwt: github: mosop/jwt version: ">= 2.0.0" # Allow versions 2.0.0 and greater """ **Don't Do This:** Use vague version specifiers like "*" or no version specifier at all. This can lead to unpredictable dependency resolution. Avoid overly strict version pinning unless absolutely necessary due to compatibility issues. ### 1.3. Keeping Dependencies Up-to-Date **Do This:** Regularly update your dependencies using "shards update". Review the changelogs of updated dependencies for potential breaking changes and adjust your code accordingly. **Why:** Keeping dependencies up-to-date ensures that you benefit from bug fixes, performance improvements, and security patches. Reviewing changelogs helps you proactively address any compatibility issues introduced by updates. **Example:** """bash # Update all dependencies shards update # Update a specific shard shards update kemal """ **Don't Do This:** Neglect dependency updates for extended periods. This increases the risk of security vulnerabilities and compatibility issues. Also, blindly run "shards update" without reviewing changelogs. ### 1.4 Private Shards and Repositories **Do This:** When working with private dependencies, use appropriate authorization mechanisms (e.g., SSH keys, access tokens) and ensure that your deployment environment has the necessary permissions to access the private repositories. Store credentials safely using environment variables or dedicated secrets management tools. **Why:** Secure access management is critical for protecting sensitive code held in private repositories. Following a least-privilege principle minimizes the risk of unauthorized access to confidential dependencies. **Example:** """yaml # shards.yml repository: type: git url: git@github.com:my-org/private-shard.git branch: main private: true """ **Don't Do This:** Hardcode credentials in your "shards.yml", environment variables, or source code. Grant excessive permissions for private shard access to users or environments. ## 2. Code Formatting and Linting Consistent code formatting and linting are essential for code readability and maintainability. ### 2.1. Using "crystal tool format" **Do This:** Use "crystal tool format" to automatically format your code according to the standard Crystal style. Integrate this command into your development workflow (e.g., as a pre-commit hook). **Why:** "crystal tool format" enforces a consistent code style, reducing subjective formatting debates and making the code easier to read and understand. **Example:** """bash crystal tool format src/**/*.cr """ **Don't Do This:** Rely on manual formatting only. This is time-consuming and prone to inconsistencies. ### 2.2. Using "ameba" for Linting **Do This:** Use "ameba" (a Crystal linter) to identify potential code quality issues, style violations and potential errors. Integrate "ameba" into your CI/CD pipeline to automatically check code quality on every commit or pull request. **Why:** "ameba" helps you catch common mistakes and enforce coding standards, leading to cleaner, more maintainable code. **Example:** 1. Add "ameba" as a dependency in "shards.yml": """yaml dependencies: ameba: github: veelenga/ameba version: "~> 1.0" """ 2. Run "shards install". 3. Run "ameba" to analyze your code: """bash ameba """ 4. Configure ".ameba.yml" to customize linting rules according to project conventions. **Don't Do This:** Ignore linting warnings or disable "ameba" checks. ### 2.3. Editor Integration **Do This:** Configure your code editor (e.g., VS Code, Atom, Sublime Text) to automatically format your code on save and display linting errors. Use plugins such as the Crystal VS Code extension. **Why:** Editor integration makes code formatting and linting seamless and automatic, improving developer productivity and code quality. **Example:** * **VS Code:** Install the "Crystal Language" extension and configure it to run "crystal tool format" on save. Configure "ameba" integration for real-time linting feedback. **Don't Do This:** Rely solely on command-line tools for formatting and linting. ## 3. Testing Frameworks and Strategies Comprehensive testing is crucial for ensuring the reliability of your Crystal code. ### 3.1. Using the Built-in "spec" Framework **Do This:** Use the built-in "spec" framework for writing unit tests and integration tests. **Why:** "spec" provides a simple and expressive way to define tests in Crystal. **Example:** """crystal # spec/my_class_spec.cr require "spec" require "../src/my_class" describe MyClass do it "should return 42" do my_object = MyClass.new my_object.answer.should eq 42 end end """ **Don't Do This:** Neglect writing tests or use ad-hoc testing methods. ### 3.2. Test-Driven Development (TDD) **Do This:** Consider adopting a TDD approach, writing tests before writing the actual code. **Why:** TDD helps you clarify requirements and design better APIs. It also ensures that your code is testable from the start. **Example:** 1. Write a failing test for a feature """crystal # spec/calculator_spec.cr require "spec" require "../src/calculator" describe Calculator do it "should add two numbers correctly" do calc = Calculator.new calc.add(2, 3).should eq 5 end end """ 2. Implement only the necessary code to pass the above test. """crystal # src/calculator.cr class Calculator def add(a : Int32, b : Int32) : Int32 a + b end end """ 3. Refactor **Don't Do This:** Write code without tests, or write tests only after the code is complete. ### 3.3. Using Mocks and Stubs **Do This:** Use mocks and stubs to isolate units of code during testing. Libraries like "test_double" can assist. **Why:** Mocks and stubs allow you to test code in isolation, without relying on external dependencies or databases. **Example:** """crystal # spec/my_service_spec.cr require "spec" require "../src/my_service" require "test_double" describe MyService do it "should call the external API" do api_mock = mock(ExternalApi) expect(api_mock).to receive(:get_data).and_return("mocked data") service = MyService.new(api_mock.object) result = service.process_data result.should eq "processed mocked data" end end """ **Don't Do This:** Rely heavily on real external dependencies in your tests. ### 3.4. Integration Testing **Do This:** Write integration tests to verify that different parts of your application work correctly together. Test interactions with databases, external APIs, and other systems. **Why:** Integration tests catch issues that unit tests might miss, such as incorrect data mappings or network connectivity problems. **Example:** Write integration tests to see if kemal web application correctly interacts with database. This might require accessing the real database. **Don't Do This:** Only perform unit testing and ignore integration testing entirely. ## 4. Logging and Monitoring Proper logging and monitoring are essential for debugging and identifying issues in production. ### 4.1. Using the Standard Library "log" **Do This:** Use the standard library "log" module for logging. Configure different log levels (e.g., "DEBUG", "INFO", "WARN", "ERROR") to control the verbosity of your logs. **Why:** The "log" module provides a simple and standard way to log messages in Crystal. **Example:** """crystal require "log" Log.info "Application started" Log.debug "Processing request: #{request.inspect}" Log.warn "High CPU usage detected" """ **Don't Do This:** Use "puts" or "print" for logging in production code. These methods are not configurable and can clutter the output. ### 4.2. Structured Logging **Do This:** Use structured logging formats (e.g., JSON) to make logs easier to parse and analyze. Consider using a library like "logstash" with appropriate configurations. **Why:** Structured logs enable you to easily filter, aggregate, and analyze logs using tools like Elasticsearch, Graylog, or Splunk. **Example:** """crystal require "json" require "log" data = { "event": "user_login", "user_id": 123, "timestamp": Time.utc } Log.info(data.to_json) """ **Don't Do This:** Log unstructured text only. This makes it difficult to automate log analysis. ### 4.3. Centralized Logging **Do This:** Configure your application to send logs to a centralized logging system (e.g., ELK stack, Graylog, Papertrail). **Why:** Centralized logging makes it easier to monitor and analyze logs from multiple instances of your application. **Example:** Use Logstash in combination with Elasticsearch and Kibana to collect, process and visualize logs from multiple Crystal app instances. **Don't Do This:** Rely solely on local log files. This makes it difficult to troubleshoot issues in distributed environments. ### 4.4. Monitoring Tools **Do This:** Integrate your application with monitoring tools (e.g., Prometheus, Grafana, Datadog) to track key performance metrics (e.g., CPU usage, memory usage, response time). **Why:** Monitoring tools provide real-time insights into the health and performance of your application. **Example:** Use Prometheus client libraries to expose application metrics and Grafana for dashboard visualization. **Don't Do This:** Operate applications in production without adequate monitoring. ## 5. Concurrency and Parallelism Crystal provides powerful concurrency features. Using them correctly is important for performance. ### 5.1. Using Fibers **Do This:** Use fibers for concurrent operations that are I/O-bound (e.g., network requests, database queries). **Why:** Fibers are lightweight and efficient, allowing you to handle many concurrent operations without creating excessive overhead. Crystal's scheduler will correctly manage fiber execution. **Example:** """crystal require "http/client" def fetch_url(url) Fiber.new do response = HTTP::Client.get(url) puts "Fetched #{url}: #{response.status_code}" end.resume end urls = ["https://crystal-lang.org", "https://kemalcr.com", "https://www.google.com"] urls.each { |url| fetch_url(url) } sleep 1 # Wait for fibers to complete """ **Don't Do This:** Use threads for I/O-bound operations. Threads are heavier than fibers and may not scale as well. ### 5.2. Using Channels **Do This:** Use channels for communicating between fibers safely and efficiently. **Why:** Channels provide a way to send data between fibers without risking race conditions or data corruption. **Example:** """crystal channel = Channel(String).new Fiber.new do channel.send("Hello from Fiber 1") end.resume Fiber.new do message = channel.receive puts "Fiber 2 received: #{message}" end.resume sleep 1 # Wait for fibers to complete """ **Don't Do This:** Share mutable data between fibers without proper synchronization. ### 5.3. Parallel Processing **Do This:** Use parallel processing for CPU-bound operations to take advantage of multiple cores. **Why:** Parallel processing can significantly improve performance for computationally intensive tasks. Crystal supports parallel processing with the "spawn" keyword. **Example:** """crystal def process_data(data : Array(Int32)) : Array(Int32) data.map { |x| x * 2 } end data = (1..1000).to_a # Naive implementation result = process_data(data) # Parallelized implementation result_parallel = Parallel.map(data) { |x| x * 2 } """ **Don't Do This:** Over-parallelize operations. This can lead to increased overhead and decreased performance. Measure and profile to find the optimal level of parallelism. Avoid unnecessary data copying during parallel operations. ## 6. Tooling for Specific Frameworks ### 6.1 Kemal When devoloping web application in Crystal, Kemal is often choice. **Do This:** Use Kemal CLI tools for scaffolding new projects and generating code. Understand Kemal's middleware architecture and use it to structure request processing logic. **Example:** """bash kemal new my_app cd my_app kemal serve """ **Don't Do This:** Reinvent the wheel by not using Kemal's built-in features for common tasks. ### 6.2 ORM (Object-Relational Mapping) **Do This:** Use an ORM like Granite or Crinja for database interactions instead of writing raw SQL queries. **Why:** ORMs abstract away database-specific details, improve code readability, and reduce the risk of SQL injection vulnerabilities. **Example (Granite):** """crystal require "granite" require "pg" Granite::Base.connection_pool = DB::Pool.new(ENV["DATABASE_URL"]) record User do field :id, PrimaryKey field :name, String field :email, String end user = User.new(name: "Alice", email: "alice@example.com") user.save # => true """ **Don't Do This:** Write raw SQL queries directly unless absolutely necessary for performance reasons or complex queries. ## 7. API Design and Versioning ### 7.1. Follow RESTful Principles **Do This:** Design APIs following RESTful principles. **Why:** RESTful APIs are easy to understand, predictable, and scalable. **Example:** * Use standard HTTP methods (GET, POST, PUT, DELETE). * Use nouns for resources (e.g., "/users", "/products"). * Use HTTP status codes to indicate success or failure. * Use hypermedia links (HATEOAS) to enable API discovery. **Don't Do This:** Create ad-hoc API designs that deviate from RESTful conventions. ### 7.2. API Versioning **Do This:** Use API versioning to maintain backward compatibility when introducing breaking changes. Use the header or URL versioning strategy **Why:** API versioning allows you to evolve your API without breaking existing clients. **Example (header versioning):** * Include a "version" parameter in the header of the request **Example (URL versioning):** * Prefix API endpoints with a version number (e.g., "/v1/users", "/v2/users"). **Don't Do This:** Make breaking API changes without versioning. Blindly update API without backward compatibility. ### 7.3. Documentation **Do This:** Document your APIs using tools like Swagger/OpenAPI. **Why:** API documentation makes it easier for developers to understand and use your APIs. **Example:** Use the openapi generator toolchain to automatically generate API clients and server stubs from OpenAPI specifications. **Don't Do This:** Neglect API documentation, or rely on incomplete or outdated documentation. ## 8. Community and Learning ### 8.1. Staying Up-to-Date **Do This:** Follow the official Crystal blog, release notes, and community forums to stay up-to-date with the latest developments in the language and ecosystem. Actively participate in the Crystal community on platforms like GitHub, Gitter, and Discourse to learn from others and contribute your knowledge. **Why:** Keeping current with the Crystal language and its evolving ecosystem ensures that you are using the latest features, best practices, and security updates. Community engagement provides opportunities for collaborative learning, knowledge sharing, and professional growth. **Don't Do This:** Rely solely on outdated tutorials or documentation. Avoid community discussions and feedback. ### 8.2 Contributing **Do This:** Consider contributing to open-source Crystal projects, whether by submitting bug fixes, suggesting new features, or improving documentation. Share your experience and knowledge with other developers through blog posts, articles, or conference talks. Promote Crystal within your organization and within the broader technology community. **Why:** Contributing to open-source projects helps improve the Crystal ecosystem for everyone. Sharing your knowledge and expertise fosters a culture of learning and innovation. Promoting Crystal helps to expand its user base and attract further investment in the language and its tooling. **Don't Do This:** Hoard your knowledge and contributions. Avoid participating in the Crystal community or promoting the language. This document provides a comprehensive set of tooling and ecosystem standards for Crystal development, which can significantly improve code quality, maintainability, and performance. By adhering to these guidelines, development teams can build robust and scalable applications.

# Security Best Practices Standards for Crystal This document outlines security best practices for Crystal development, serving as a guide for developers and AI coding assistants. It focuses on protecting against common vulnerabilities and implementing secure coding patterns specific to Crystal, embracing modern approaches based on the latest version. ## 1. Input Validation and Sanitization ### Standard Validate and sanitize all external input to prevent injection attacks (SQL injection, XSS, command injection). ### Why Untrusted data from users or external systems can corrupt data, execute malicious code, or compromise the system. ### Guidelines * **Do This:** * Always validate input data types, lengths, ranges, and formats against expected values. * Use prepared statements and parameterized queries for database interactions. * Encode output to prevent XSS vulnerabilities. * Use regular expressions or dedicated libraries for input validation. * **Don't Do This:** * Trust user input without validation. * Concatenate user input directly into SQL queries. * Display user-provided data directly without encoding. * Rely solely on client-side validation. ### Code Examples **SQL Injection Prevention (Prepared Statements)** """crystal require "db" DB.open("sqlite3:mydb.db") do |db| # Do This: Use prepared statements statement = db.prepare("SELECT * FROM users WHERE username = ? AND password = ?") username = "user1" password = "password123" result_set = statement.execute(username, password) result_set.each do |row| puts "User found: #{row}" end # Don't Do This: Avoid string interpolation # username = "user1; DROP TABLE users;" # Malicious input # query = "SELECT * FROM users WHERE username = '#{username}'" # Vulnerable! # db.query(query).each do |row| # puts "User found: #{row}" # end end """ **XSS Prevention (Output Encoding)** """crystal require "kemal" get "/" do |env| user_input = env.params["query"] || "Default Value" # Assume "query" parameter contains user input. # Do This: Escape HTML entities to prevent XSS escaped_input = CGI.escape_html(user_input) "<h1>You searched for: #{escaped_input}</h1>" end """ **Command Injection Prevention** """crystal # Do This: Use shell-safe methods or avoid shell commands whenever possible. require "process" def execute_safe_command(file_path : String) # Use Process::run to avoid shell interpretation of arguments result = Process.run("ls", [file_path]) if result.success? puts result.output else puts result.error end end # Example usage with a safe file path execute_safe_command("/safe/path/to/file.txt") # Don't Do This: Directly pass unsanitized user input to shell commands # def execute_command(user_input : String) # "ls #{user_input}" # Highly vulnerable to command injection # end # Example of VERY UNSAFE usage (demonstration purposes only!) # execute_command("$(rm -rf /)") # Example of malicious input """ ### Anti-Patterns * String interpolation directly in SQL queries. * Instead, always use prepared statements. * Trusting client-side validation without server-side verification. * Client-side validation is easily bypassed. * Displaying raw user input without encoding. * This can lead to XSS vulnerabilities. ## 2. Authentication and Authorization ### Standard Implement robust authentication and authorization mechanisms to identify and control user access. ### Why Weak authentication and authorization can lead to unauthorized access, data breaches, and compromised functionality. ### Guidelines * **Do This:** * Use strong password hashing algorithms (bcrypt, scrypt, argon2). * Implement multi-factor authentication (MFA). * Enforce principle of least privilege. * Use established authentication/authorization libraries (e.g., JWT, OAuth). * **Don't Do This:** * Store passwords in plain text. * Use weak hashing algorithms (MD5, SHA1). * Grant excessive permissions to users. * Implement custom authentication/authorization schemes without thorough security review. ### Code Examples **Password Hashing with bcrypt** """crystal require "bcrypt" def hash_password(password : String) : String BCrypt::Password.create(password) end def verify_password(password : String, hashed_password : String) : Bool BCrypt::Password.new(hashed_password) == password end # Example usage password = "mysecretpassword" hashed_password = hash_password(password) puts "Hashed password: #{hashed_password}" # Store the 'hashed_password' in database # Verification if verify_password("mysecretpassword", hashed_password) puts "Authentication successful!" else puts "Authentication failed!" end """ **JWT Authentication (using "jwt" shard)** """crystal require "jwt" require "time" SECRET_KEY = "your_secret_key" # Replace with a strong, randomly generated key def generate_token(user_id : Int32) : String payload = { "user_id" => user_id, "exp" => (Time.utc + 3600).to_unix # Token expiration time (1 hour) } JWT.encode(payload, SECRET_KEY, "HS256") end def verify_token(token : String) : Hash(String, JSON::Any)? begin decoded_token = JWT.decode(token, SECRET_KEY, true, {algorithm: "HS256"}) decoded_token[0].as_h? # Return the payload as a hash if verification is successful rescue JWT::DecodeError => e puts "Token verification failed: #{e.message}" nil end end # Example Usage user_id = 123 token = generate_token(user_id) puts "Generated token: #{token}" decoded = verify_token(token) if decoded puts "Authenticated User ID: #{decoded["user_id"]}" else puts "Authentication failed." end """ **Authorization (Role-Based Access Control)** """crystal enum Role Admin Editor Viewer end record User, id : Int32, role : Role def authorize(user : User, required_role : Role) : Bool user.role <= required_role end # Example Usage user = User.new(1, Role::Editor) if authorize(user, Role::Admin) puts "User is authorized to perform admin actions." elsif authorize(user, Role::Editor) puts "User is authorized to perform editor actions." else puts "User has insufficient privileges." end """ ### Anti-Patterns * Using predictable or easily guessable passwords. * Creating custom authentication schemes without rigorous security review. * Storing sensitive information in cookies without proper encryption and security flags. * Relying solely on front-end authorization. ## 3. Data Encryption and Storage ### Standard Encrypt sensitive data at rest and in transit to protect confidentiality. ### Why Data encryption prevents unauthorized access to sensitive information if systems are compromised. ### Guidelines * **Do This:** * Use TLS/SSL for all network communication. * Encrypt sensitive data at rest using strong encryption algorithms (AES). * Properly manage encryption keys and secrets. * Use secure storage solutions for sensitive data. * **Don't Do This:** * Transmit sensitive data over unencrypted channels (HTTP). * Store encryption keys in the code or version control. * Use weak or outdated encryption algorithms. ### Code Examples **TLS/SSL (using Kemal)** """crystal require "kemal" # Kemal framework defaults to TLS/SSL when running in production. # Ensure proper configuration for SSL certificates (e.g., using Let's Encrypt). # Configure Kemal to bind to HTTPS port (443 by default) and use your SSL certificates # In Kemal, ensure SSL certificates are properly configured via environment variables or command-line arguments. # KEMAL_SSL_CERT=path/to/your/certificate.pem # KEMAL_SSL_KEY=path/to/your/private_key.pem get "/" do |env| "Welcome to my secure site!" end """ **Data Encryption with AES (using "OpenSSL" binding)** """crystal require "openssl" KEY = "1234567890123456" # a 16-byte key for AES-128 IV = "abcdefghijklmnop" # Initialization Vector (must be 16 bytes for AES) def encrypt(data : String, key : String, iv : String) : String cipher = OpenSSL::Cipher.new("aes-128-cbc") cipher.encrypt cipher.key = key cipher.iv = iv encrypted = cipher.update(data) + cipher.final Base64.encode64(encrypted) end def decrypt(encrypted_data : String, key : String, iv : String) : String decipher = OpenSSL::Cipher.new("aes-128-cbc") decipher.decrypt decipher.key = key decipher.iv = iv decoded_data = Base64.decode64(encrypted_data) decrypted = decipher.update(decoded_data) + decipher.final decrypted end # Example Usage data = "Sensitive data to encrypt." encrypted_data = encrypt(data, KEY, IV) puts "Encrypted data: #{encrypted_data}" decrypted_data = decrypt(encrypted_data, KEY, IV) puts "Decrypted data: #{decrypted_data}" """ **Secure Secret Management (using environment variables)** """crystal # Do This: Use environment variables to store secrets db_password = ENV["DATABASE_PASSWORD"] if db_password.nil? puts "Error: DATABASE_PASSWORD environment variable not set!" exit(1) end puts "Connecting to database with password from environment." # Use db_password to connect to the database # DB.open("postgres://user:#{db_password}@host:port/database") do |db| ... end # Don't Do This: Hardcoding sensitive information or storing it in version control. #DB_PASSWORD = "hardcoded_password" # Insecure! """ ### Anti-Patterns * Storing encryption keys directly in the code. * Using weak or outdated encryption algorithms. * Transmitting sensitive data over unencrypted channels. * Failing to properly rotate encryption keys. ## 4. Error Handling and Logging ### Standard Implement proper error handling and logging mechanisms to identify and address security vulnerabilities. ### Why Detailed logs provide critical information for identifying and investigating security incidents. Careless error handling can leak sensitive information. ### Guidelines * **Do This:** * Log all security-related events (authentication failures, authorization violations, etc.). * Use structured logging formats (JSON) for easier analysis. * Implement centralized logging and monitoring. * Sanitize sensitive data before logging. * Handle exceptions gracefully and avoid revealing sensitive information in error messages. * **Don't Do This:** * Log sensitive information (passwords, API keys). * Use generic error messages that provide limited context. * Ignore exceptions without logging. ### Code Examples **Structured Logging** """crystal require "json" require "time" def log_event(level : String, message : String, data : Hash(String, String)? = nil) log_entry = { "timestamp" => Time.utc.to_s, "level" => level, "message" => message, "data" => data || {} } puts log_entry.to_json end # Example Usage log_event("INFO", "User logged in successfully", {"user_id" => "123", "ip_address" => "192.168.1.1"}) log_event("ERROR", "Authentication failed", {"username" => "testuser", "ip_address" => "192.168.1.1"}) # Sanitizing Sensitive Data sensitive_data = {"credit_card" => "1234-5678-9012-3456"} sanitized_data = {"credit_card" => "REDACTED"} log_event("WARN", "Suspicious activity detected", sanitized_data) # Log sanitized data """ **Error Handling** """crystal def safe_division(a : Int32, b : Int32) : Int32? begin a // b rescue DivisionByZero puts "Error: Division by zero!" log_event("ERROR", "Division by zero attempted", {"value_a" => a.to_s, "value_b" => b.to_s}) nil #Return nil to indicate failure end end # Example Usage result = safe_division(10, 0) if result puts "Result: #{result}" else puts "Operation failed." end def risky_operation(filename : String) begin file = File.open(filename) puts file.read_all file.close rescue e : Errno::ENOENT log_event("ERROR", "File not found", {"filename" => filename}) puts "File not found. Check logs for details." # User-friendly message rescue => e log_event("ERROR", "Generic error: #{e.message}", {"filename" => filename, "error" => e.class.name}) puts "An unexpected error occurred. Check logs for details." # Generic Message. end end risky_operation("non_existent_file.txt") #Example usage. """ ### Anti-Patterns * Logging sensitive information in plain text. * Using generic error messages that provide limited insight. * Over-logging, which can impact performance and storage. * Not having a central logging or alerting mechanism. ## 5. Dependencies and Vulnerability Management ### Standard Regularly update dependencies and scan for vulnerabilities to mitigate risks from third-party libraries. ### Why Vulnerabilities in dependencies can be exploited to compromise the application. ### Guidelines * **Do This:** * Use a dependency management tool (e.g., shards) to manage dependencies. * Regularly update dependencies to the latest versions. * Use security scanning tools to identify vulnerabilities in dependencies (e.g., "bundler-audit" equivalent). * Monitor security advisories for known vulnerabilities. * **Don't Do This:** * Use outdated or unsupported dependencies. * Ignore security advisories for dependencies. * Fail to regularly scan for vulnerabilities. ### Code Examples **Using Shards for Dependency Management** """yaml # shard.yml name: myapp version: 0.1.0 dependencies: kemal: github: kemalcr/kemal version: "~> 1.4" # Specify desired version. Use "~>" for compatible updates. jwt: github: joewandy/crystal-jwt version: "~> 0.5" license: MIT """ To install dependencies: """bash shards install """ To update dependencies to the latest compatible version: """bash shards update """ **Vulnerability Scanning Considerations** Crystal doesn't have a dedicated "bundler-audit" equivalent directly. Here are ways to approach this based on available information and tools. You can use these in combination for best results: 1. **Manual Review and Monitoring:** * Regularly check for security advisories related to the shards you are using. Monitor the GitHub repositories of your dependencies for reported issues and updates. * Subscribe to security mailing lists or RSS feeds that announce vulnerabilities in software libraries. 2. **Automated Vulnerability Scanning (Indirectly):** * **Container Scanning:** If you deploy your Crystal application within a Docker container, use container image scanning tools like Clair, Snyk, or Anchore to identify vulnerabilities in the base OS image and any system-level dependencies. * **CI/CD Integration:** Integrate security scanning into your CI/CD pipeline by running container scans during the build process. Fail the build if critical vulnerabilities are found. 3. **Shard-Specific Auditing (Future Tooling):** * The Crystal community may develop shard-specific auditing tools in the future. Keep an eye on community projects and announcements. Such tools would ideally analyze "shard.lock" and compare dependency versions against a known vulnerability database. **Example - Docker usage for scanning** """dockerfile FROM crystallang/crystal:latest WORKDIR /app COPY shard.yml shard.yml RUN shards install COPY . . # Dummy command to make the image scannable, replace with actual build/run execution CMD ["echo", "Application is built"] """ **Then use security scanners like "Trivy" to scan the docker image** """bash docker build -t my-crystal-app . trivy image my-crystal-app """ ### Anti-Patterns * Using outdated dependencies without updates. * Ignoring security advisories for dependencies. * Failing to scan for vulnerabilities in dependencies. * Vendoring dependencies without proper management. ## 6. Denial of Service (DoS) Protection ### Standard Implement measures to protect against denial-of-service (DoS) and distributed denial-of-service (DDoS) attacks. ### Why DoS attacks can overwhelm the system and make it unavailable to legitimate users. ### Guidelines * **Do This:** * Implement rate limiting to prevent abuse. * Use a web application firewall (WAF) to filter malicious traffic (Cloudflare, AWS WAF). * Implement request size limits. * Use caching to reduce server load. * **Don't Do This:** * Ignore potential DoS vectors. * Fail to monitor system resources for signs of attack. * Expose unnecessary endpoints. ### Code Examples **Rate Limiting (using Kemal middleware)** """crystal require "kemal" require "redis" # or any other caching system # Simple in-memory rate limiter (for demonstration; use Redis in production) RATE_LIMITS = {} of String => Int32 def rate_limit(env : HTTP::Env, redis : Redis) ip_address = env.remote_address.to_s key = "ratelimit:#{ip_address}" limit = 100 # requests per minute expiry = 60 # seconds # Atomic operation to increment the counter in Redis and get the value count = redis.incr(key) # Set expiry only if the key is new redis.expire(key, expiry) if count == 1 if count > limit env.response.status_code = 429 env.response.body = "Too Many Requests" env.response.headers["Retry-After"] = expiry.to_s return false # Halt the request else return true # Proceed with the request end end before do |env| redis = Redis.new("redis://127.0.0.1:6379") # Replace with your redis connection string. unless rate_limit(env, redis) halt # Stop processing the request if rate limit is exceeded end end get "/" do |env| "Hello, world!" end """ **Request Size Limits (Kemal)** """crystal require "kemal" #This configuration applies to all routes, can be configured at individual route levels. configure do |settings| settings.max_request_size = 10_000_000 # 10MB request limit end post "/upload" do |env| # By default Kemal already protects against large uploads due to the max_request_size configured. file = env.params.file("myfile") if file # Process the uploaded file puts "File name: #{file.filename}" puts "File size: #{file.tempfile.size}" "File uploaded successfully" else env.response.status_code = 400 "No file uploaded" end end """ ### Anti-Patterns * Not implementing any rate limiting. * Using overly permissive rate limits. * Failing to monitor for DoS attacks. * Relying solely on application-level defenses without network-level protection. ## 7. Session Management ### Standard Implement secure session management practices to protect user sessions from hijacking and unauthorized access. ### Why Vulnerable session management can lead to user account compromise. ### Guidelines * **Do This:** * Use strong, randomly generated session IDs * Regenerate the session ID after successful login * Set appropriate session expiration times * Store session data server-side * Use secure, HTTP-only cookies * Implement session timeout and idle timeout * Validate session data on each request * **Don't Do This:** * Use predictable session IDs * Store sensitive information in session cookies * Use excessively long session expiration times * Allow session fixation vulnerabilities. ### Code Examples **Secure Session Management (using Kemal middleware)** """crystal require "kemal" require "session" # configure session settings, use secure cookies, and use suitable secret. configure do |settings| settings.session_secret = "A_Very_Long_And_Random_Secret_Key" settings.session_store = Session::CookieStore.new( cookie_options: { secure: true, # Only transmit over HTTPS httponly: true, # Prevent client-side JavaScript access same_site: :strict # Helps prevent CSRF attacks } ) end before do |env| env.session # Accessing env.session initializes the session end get "/" do |env| session = env.session session["visits"] = (session["visits"] || 0).as(Int32) + 1 "Visits: #{session["visits"]}" end get "/login" do |env| # Simulating authentication successfully after submitted credentials via POST to /login env.session["user_id"] = 123 # Authenticated user ID # DO THIS: Regenerate session ID after login to prevent session fixation env.session.regenerate redirect "/" end get "/logout" do |env| env.session.clear # Clear all session data redirect "/" end """ ### Anti-Patterns * Using sequential or predictable session IDs. * Storing session IDs in the URL. * Using the same session ID for the entire session lifetime (without regeneration after login). * Using excessively long session lifetimes. * Not invalidating sessions on logout. * Using weak session secrets ## 8. File Uploads ### Standard Implement secure file upload handling to prevent malicious file uploads and potential security vulnerabilities. ### Why Unvalidated file uploads can lead to remote code execution, XSS, or denial of service attacks. ### Guidelines * **Do This:** * Validate file type based on content, not just the file extension * Sanitize filenames to prevent directory traversal attacks * Limit file sizes * Store uploaded files outside the web root * Use unique file names to prevent overwrites * Scan uploaded files for malware * Set restrictive permissions on uploaded files * **Don't Do This:** * Trust the file extension (e.g., "*.jpg", "*.png") * Store uploaded files directly in the web root * Allow arbitrary file names * Execute uploaded files directly * Use filename filtering as the only means of validation. ### Code Example """crystal require "kemal" PUBLIC_UPLOAD_DIR = "uploads" Dir.mkdir_p PUBLIC_UPLOAD_DIR # Create the directory if it doesn't exist post "/upload" do |env| file = env.params.file("myfile") if file filename = file.filename tempfile = file.tempfile # This gives you a file-like object # Validate File type based on content (MIME Type verification) instead of just extension mime_type = "file --mime-type -b #{tempfile.path}".strip # Requires 'file' utility allowed_mime_types = ["image/jpeg", "image/png", "image/gif"] unless allowed_mime_types.includes?(mime_type) env.response.status_code = 400 puts "Invalid file type: #{mime_type}" "Invalid file type." next end #Sanitize filename to prevent path traversal. sanitized_filename = filename.gsub(%r{[^a-zA-Z0-9._-]}) { "" } # Remove any characters that aren't letters, numbers, underscores, periods, or hyphens safe_filename = File.join(PUBLIC_UPLOAD_DIR, "#{Time.utc.to_unix}_#{sanitized_filename}") #Limit file uploads size. max_file_size = 10_000_000 #10MB if file.tempfile.size > max_file_size "File is too large" env.response.status_code = 413 next end # Save File File.copy(file.tempfile.path, safe_filename) puts "File uploaded to: #{safe_filename}" "File uploaded successfully." else env.response.status_code = 400 "No file uploaded" end end """ ### Anti-Patterns * Trusting client-provided filenames or MIME types without server-side validation. * Storing uploaded files in the webserver's document root without protecting them. * Allowing unrestricted file sizes or file types. * Not sanitizing uploaded filenames. * Attempting to execute uploaded files directly.

# Component Design Standards for Crystal This document outlines component design standards for Crystal projects, aiming to foster reusable, maintainable, and performant code. These standards are tailored specifically for Crystal's features and ecosystem. ## 1. Component Architecture A component is a self-contained, reusable unit of functionality with a well-defined interface. In Crystal, components can be represented by modules, classes, structs, or even standalone functions. The overall application architecture should be component-based to promote modularity. ### 1.1. Loose Coupling * **Do This:** Design components with minimal knowledge of each other. Use interfaces or abstract classes to communicate. * **Don't Do This:** Create tight dependencies between components, making them difficult to change or reuse independently. * **Why:** Loose coupling enhances modularity and flexibility. Changes in one component are less likely to affect others. """crystal # Good: Using an interface abstract class PaymentProcessor abstract def process_payment(amount : Float64) end class StripeProcessor < PaymentProcessor def process_payment(amount : Float64) puts "Processing #{amount} with Stripe" end end class PaypalProcessor < PaymentProcessor def process_payment(amount : Float64) puts "Processing #{amount} with Paypal" end end # The client depends on the abstract PaymentProcessor, not the concrete implementations class Order def initialize(@payment_processor : PaymentProcessor) end def checkout(amount : Float64) @payment_processor.process_payment(amount) end end stripe = StripeProcessor.new order = Order.new(stripe) order.checkout(100.0) """ """crystal # Bad: Tightly Coupled class StripePayment def process(amount : Float64) puts "Processing payment using Stripe: #{amount}" end end class OrderProcessor def process_order(amount : Float64) stripe = StripePayment.new stripe.process(amount) # Tight coupling - OrderProcessor knows too much about StripePayment end end order_processor = OrderProcessor.new order_processor.process_order(50.0) """ ### 1.2. High Cohesion * **Do This:** Ensure each component encapsulates a single, well-defined purpose. * **Don't Do This:** Create God classes or modules that try to handle too many unrelated responsibilities. * **Why:** High cohesion leads to simpler, more understandable code. It simplifies maintenance and testing. """crystal # Good: High Cohesion module UserAuthentication def self.authenticate(username : String, password : String) : Bool # ... authentication logic ... true # Stubbed for example end def self.authorize(user_id : Int32, resource : String) : Bool # ... authorization logic ... true # Stubbed for example end end # The module is focused solely on user authentication and authorization """ """crystal # Bad: Low Cohesion module UserManagement def self.authenticate(username : String, password : String) : Bool # ... authentication logic ... true # Stubbed for example end def self.create_user(username : String, email : String) # ... user creation logic ... end def self.send_welcome_email(email : String) # ... send welcome email logic ... end end # The module attempts to handle authentication, user creation, and email sending - low cohesion! """ ### 1.3. Explicit Interfaces * **Do This:** Define clear interfaces for each component using abstract classes or modules with defined public methods. * **Don't Do This:** Rely on implicit interfaces or expose internal implementation details. * **Why:** Explicit interfaces promote predictability and allow for easier substitution of components. """crystal # Good: Interface definition using abstract class abstract class Storage abstract def save(key : String, value : String) abstract def load(key : String) : String? # String? to indicate nullable return abstract def delete(key : String) end class FileStorage < Storage def save(key : String, value : String) File.write(key, value) end def load(key : String) : String? File.read(key) rescue nil end def delete(key : String) File.delete(key) end end # Usage: storage = FileStorage.new storage.save("my_key", "my_value") puts storage.load("my_key") """ """crystal # Bad: Implicit Interface class DataWriter def write_data(data) # Implementation implies it takes any data puts "Writing data: #{data}" end end class DataProcessor def process(writer) writer.write_data({"key" => "value"}) # Assumes DataWriter has write_data, but no explicit contract. end end """ ## 2. Component Implementation ### 2.1. Use Crystal Generics Judiciously * **Do This:** Use generics to create reusable components that work with different types. * **Don't Do This:** Overuse generics, leading to complex and unreadable code. Consider type constraints. * **Why:** Generics enhance code reuse while maintaining type safety. """crystal # Good: Using Generics with Type Constraints class Cache(T) getter data : Hash(String, T) def initialize @data = Hash(String, T).new end def put(key : String, value : T) @data[key] = value end def get(key : String) : T? @data[key] end end cache_string = Cache(String).new cache_string.put("name", "Crystal") puts cache_string.get("name") cache_int = Cache(Int32).new cache_int.put("port", 8080) puts cache_int.get("port") """ """crystal # Bad: Overusing Generics class GenericProcessor(T, U, V) # Too many generic types, hard to reason about def process(input1 : T, input2 : U) : V # Complex logic involving T, U, and V... input1.to_s + input2.to_s # Just an example end end """ ### 2.2. Favor Composition over Inheritance * **Do This:** Build complex components by combining simpler components rather than relying heavily on inheritance. * **Don't Do This:** Create deep inheritance hierarchies that become rigid and difficult to maintain. * **Why:** Composition provides better flexibility and avoids the problems associated with the fragile base class problem. """crystal # Good: Composition example class Logger def log(message : String) puts "Log: #{message}" end end class StatsCollector def collect_stats(data) puts "Collecting Stats: #{data}" end end class Service def initialize(@logger : Logger, @stats : StatsCollector) end def run(data) @logger.log("Starting service") @stats.collect_stats(data) puts "Service running with #{data}" @logger.log("Service completed") end end logger = Logger.new stats = StatsCollector.new service = Service.new(logger, stats) service.run({"requests" => 100}) """ """crystal # Bad: Deep Inheritance class BaseComponent def initialize puts "Base Initialized" end end class ExtendedComponent < BaseComponent def initialize super # Always remember to call super! puts "Extended initialized" end end class EvenMoreExtended < ExtendedComponent def initialize super # Remember to call super! puts "EvenMoreExtended Initialized" end end component = EvenMoreExtended.new # Difficult to trace initialization flow. Inheritance makes debugging harder. """ ### 2.3. Use Modules for Namespaces and Utility Functions * **Do This:** Group related functions and constants within modules to avoid naming conflicts and organize code. * **Don't Do This:** Define global functions and constants that pollute the namespace. Modules offer a better approach. * **Why:** Modules create clear boundaries and improve code organization. """crystal # Good: Module Usage module StringUtils def self.reverse(str : String) : String str.reverse end def self.capitalize(str : String) : String str.capitalize end end puts StringUtils.reverse("hello") # olleh puts StringUtils.capitalize("world") # World """ """crystal # Bad: Global functions def reverse_string(str : String) : String str.reverse end def capitalize_string(str : String) : String str.capitalize end puts reverse_string("hello") puts capitalize_string("world") # pollute the global namespace """ ### 2.4. Immutability Where Possible * **Do This:** Favor immutable data structures and components to avoid unexpected side effects. * **Don't Do This:** Mutate data structures directly whenever possible. * **Why:** Immutability simplifies reasoning about code and improves thread safety. """crystal # Good: Immutability record Point, x : Int32, y : Int32 point1 = Point.new(1, 2) point2 = Point.new(point1.x + 1, point1.y + 1) # Creating new instance, not mutating puts point1 puts point2 """ """crystal # Bad: Mutable State class MutableCounter property count : Int32 def initialize @count = 0 end def increment @count += 1 # Mutates the state directly end end counter = MutableCounter.new counter.increment puts counter.count """ ### 2.5. Error Handling Strategy * **Do This:** Employ exceptions and "Result" types to deal with errors gracefully. * **Don't Do This:** Ignore errors or rely on return codes without proper handling. * **Why:** Robust error handling prevents application crashes and allows for graceful recovery. """crystal # Good: Using "Result" type for error handling require "option_parser" def safe_divide(a : Float64, b : Float64) : Result(Float64, String) if b == 0.0 return Error("Cannot divide by zero") else return Ok(a / b) end end result = safe_divide(10.0, 2.0) if result.success? puts "Result: #{result.value}" else puts "Error: #{result.error}" end result = safe_divide(5.0, 0.0) if result.success? puts "Result: #{result.value}" else puts "Error: #{result.error}" end """ """crystal # Bad: Ignoring Errors def divide(a : Float64, b : Float64) : Float64 a / b # No handling of potential division by zero end puts divide(10.0, 0.0) # Will crash the program """ ## 3. Specific Design Patterns for Crystal ### 3.1. Strategy Pattern * **How:** Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it. """crystal # Strategy Interface abstract class CompressionStrategy abstract def compress(file_path : String) end # Concrete Strategies class ZipCompression < CompressionStrategy def compress(file_path : String) puts "Compressing #{file_path} using ZIP" end end class GzipCompression < CompressionStrategy def compress(file_path : String) puts "Compressing #{file_path} using GZIP" end end # Context class FileCompressor def initialize(@strategy : CompressionStrategy) end def compress_file(file_path : String) @strategy.compress(file_path) end end # Client Code zip_strategy = ZipCompression.new gzip_strategy = GzipCompression.new compressor = FileCompressor.new(zip_strategy) compressor.compress_file("my_file.txt") compressor = FileCompressor.new(gzip_strategy) compressor.compress_file("another_file.txt") """ ### 3.2. Observer Pattern * **How:** Define a one-to-many dependency between objects so that when one object changes state, all its dependents are notified and updated automatically. """crystal # Subject interface class Subject def initialize @observers = [] of Observer end def attach(observer : Observer) @observers << observer end def detach(observer : Observer) @observers.delete(observer) end def notify @observers.each { |observer| observer.update(self) } end end # Concrete Subject class NewsPublisher < Subject getter news : String def news=(news : String) @news = news notify # Notify after state change end end # Observer Interface abstract class Observer abstract def update(subject : Subject) end # Concrete Observers class EmailSubscriber < Observer def update(subject : Subject) puts "Email Subscriber received news: #{subject.news}" end end class SMSSubscriber < Observer def update(subject : Subject) puts "SMS Subscriber received news: #{subject.news}" end end # Client Code publisher = NewsPublisher.new email_subscriber = EmailSubscriber.new sms_subscriber = SMSSubscriber.new publisher.attach(email_subscriber) publisher.attach(sms_subscriber) publisher.news = "Crystal 1.7 released!" """ ### 3.3. Factory Pattern * **How:** Define an interface for creating an object, but let subclasses decide which class to instantiate. Factory Method lets a class defer instantiation to subclasses. """crystal # Abstract Product abstract class Document abstract def open end # Concrete Products class PDFDocument < Document def open puts "Opening PDF document" end end class TextDocument < Document def open puts "Opening Text document" end end # Abstract Factory abstract class DocumentFactory abstract def create_document : Document end # Concrete Factories class PDFFactory < DocumentFactory def create_document : Document PDFDocument.new end end class TextFactory < DocumentFactory def create_document : Document TextDocument.new end end # Client Code pdf_factory = PDFFactory.new pdf_document = pdf_factory.create_document pdf_document.open text_factory = TextFactory.new text_document = text_factory.create_document text_document.open """ ## 4. Code Style and Formatting * **Use "crystal tool format":** Consistently format code using the built-in formatter. * **Follow Naming Conventions:** class "PascalCase", method "snake_case". * **Limit Line Length:** Aim for a maximum of 80-120 characters per line. * **Add Comments:** Clear and concise comments to explain complex logic. Use proper documentation for public APIs. ## 5. Conclusion Adhering to these component design standards will significantly improve the quality, maintainability, and reusability of Crystal codebases. This document will serve as a guide for new developers and a reference for maintaining consistent code quality across projects.

# Deployment and DevOps Standards for Crystal This document outlines the coding standards for deployment and DevOps practices in Crystal. It aims to guide developers in building reliable, scalable, and maintainable Crystal applications. These standards should be used with AI-assisted coding tools to improve the quality and consistency of the code produced. ## 1. Build Processes and CI/CD ### 1.1 General Principles * **Consistent Builds:** Ensure that builds are reproducible across different environments. * **Automated Pipelines:** Automate build, test, and deployment processes. * **Infrastructure as Code (IaC):** Manage infrastructure using code to enhance repeatability and traceability. ### 1.2 Standards * **Do This:** Use a build tool like "make" or "crystal Makefile". * **Why:** Provides a standard interface for building and managing your application. """makefile .PHONY: all deps build test release clean PREFIX ?= /usr/local all: deps build deps: shards install build: crystal build src/your_app.cr --release test: crystal spec install: build install -D -m 755 ./your_app $(PREFIX)/bin/your_app uninstall: rm -f $(PREFIX)/bin/your_app release: build tar czvf your_app-$(VERSION).tar.gz your_app config/application.yml clean: rm -rf ./bin/* ./lib/* .shards/ """ * **Don't Do This:** Manually compile and deploy your application. * **Why:** Increases the risk of inconsistencies and errors. ### 1.3 CI/CD Pipeline * **Do This:** Use CI/CD tools like GitHub Actions, GitLab CI, Jenkins, or CircleCI. * **Why:** Automates build, test, and deployment processes, improving efficiency and reducing errors. """yaml # .github/workflows/main.yml name: CI/CD on: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Install Crystal uses: oprypin/install-crystal@v1 with: crystal: 1.11.2 # Use a specific, recent stable version - name: Install Dependencies run: shards install - name: Build run: crystal build src/your_app.cr --release - name: Run Tests run: crystal spec - name: Create Release Archive if: github.ref == 'refs/heads/main' && github.event_name == 'push' run: | VERSION=$(git describe --tags --abbrev=0) tar czvf your_app-${VERSION}.tar.gz your_app config/application.yml mv your_app-${VERSION}.tar.gz release.tar.gz - name: Upload Release Archive if: github.ref == 'refs/heads/main' && github.event_name == 'push' uses: actions/upload-artifact@v3 with: name: release-archive path: release.tar.gz """ * **Don't Do This:** Deploy directly from your development machine. * **Why:** Introduces inconsistencies and potential security risks. ### 1.4 Containerization * **Do This:** Use Docker to build and deploy your Crystal application. * **Why:** Ensures consistent environments across different stages (development, testing, production). """dockerfile # Dockerfile FROM crystallang/crystal:latest-alpine as builder WORKDIR /app # Copy only the shard.yml and shard.lock to leverage Docker cache COPY shard.yml shard.lock ./ # Install shards dependencies RUN shards install --production COPY . . RUN crystal build src/your_app.cr --release FROM alpine:latest WORKDIR /app COPY --from=builder /app/your_app ./ COPY config/application.yml ./ EXPOSE 3000 CMD ["./your_app"] """ * **Don't Do This:** Deploy your application directly without containerization. * **Why:** Creates dependency conflicts and inconsistencies. ### 1.5 Infrastructure as Code (IaC) * **Do This:** Use tools like Terraform, Ansible, or Chef to manage infrastructure. * **Why:** Infrastructure can be versioned and managed more effectively. """terraform # Terraform example (simplified) resource "aws_instance" "app_server" { ami = "ami-xxxxxxxxxxxxxxxxx" # Replace with a valid AMI instance_type = "t2.micro" tags = { Name = "your_app-server" } } output "public_ip" { value = aws_instance.app_server.public_ip } """ * **Don't Do This:** Manually provision and configure servers. * **Why:** Error-prone and difficult to maintain. ## 2. Production Considerations ### 2.1 General Principles * **Configuration Management:** Separate configuration from code. * **Logging and Monitoring:** Implement comprehensive logging and monitoring. * **Security Hardening:** Follow security best practices. ### 2.2 Configuration Management * **Do This:** Use environment variables or configuration files (e.g., YAML, JSON) to manage application settings. * **Why:** Allows you to change settings without modifying the code. """crystal require "yaml" CONFIG_PATH = "./config/application.yml" CONFIG = YAML.parse_file(CONFIG_PATH)[ENV["CRYSTAL_ENV"]? || "development"].as_h DB_HOST = CONFIG["db_host"].as_s DB_PORT = CONFIG["db_port"].as_i """ """yaml # config/application.yml development: db_host: "localhost" db_port: 5432 production: db_host: "db.example.com" db_port: 5432 """ * **Don't Do This:** Hardcode configuration values in your application. * **Why:** Makes it difficult to manage settings across different environments. ### 2.3 Logging * **Do This:** Use a logging library like "log" or "kemal-logger" and log structured data. * **Why:** Easier to debug issues and monitor application health. """crystal require "log" Log.setup(level: Log::Severity::INFO) Log.info { "Application started" } begin # Some code that might raise an exception raise "Something went wrong" rescue e : Exception Log.error(exception: e, message: "An error occurred") end """ * **Don't Do This:** Rely solely on "puts" or "print" statements for logging. * **Why:** Difficult to manage and analyze logs in a production environment. ### 2.4 Monitoring * **Do This:** Integrate with monitoring tools like Prometheus, Grafana, DataDog, or New Relic. Use metrics and alerts. * **Why:** Gives insight into application performance and health. """crystal # Example with Prometheus (using a hypothetical client) require "prometheus" metrics = Prometheus::Registry.new http_requests_total = metrics.counter( :http_requests_total, "Total number of HTTP requests." ) get "/metrics" do |context| context.response.content_type = "text/plain; version=0.0.4; charset=utf-8" context.response.print metrics.to_text end before_all do |context| http_requests_total.increment end """ * **Don't Do This:** Neglect to monitor your application in production. * **Why:** You won't be aware of performance issues or errors. ### 2.5 Security * **Do This:** * Follow security best practices for web applications (e.g., OWASP). * Use HTTPS for all communication. * Regularly update dependencies to patch security vulnerabilities. * Implement input validation and output encoding & escaping to prevent injection attacks. * **Don't Do This:** * Store sensitive information (e.g., passwords, API keys) in code. * Expose unnecessary ports or services. * Run your application as root. ### 2.6 Handling Secrets * **Do This:** Utilize secret management tools like Vault, AWS Secrets Manager, or environment variables in secure deployment systems. * **Why:** Protect sensitive information from unauthorized access. """crystal # Example using environment variables (ensure the deployment platform supports masking sensitive env vars) DB_PASSWORD = ENV["DB_PASSWORD"] unless DB_PASSWORD Log.fatal { "DB_PASSWORD environment variable not set!" } exit(1) end # Use DB_PASSWORD to connect to the database """ * **Don't Do This:** Store secrets directly in configuration files or environment variables exposed within the code repository. * **Why:** Compromises the security of your application. ### 2.7 Zero-Downtime Deployments * **Do This:** Implement rolling deployments or blue/green deployments to minimize downtime. * **Why:** Ensures continuous availability of your application. * **How:** Uses Kubernetes deployments strategies. """yaml # Kubernetes Deployment example (rolling update) apiVersion: apps/v1 kind: Deployment metadata: name: your-app-deployment spec: replicas: 3 selector: matchLabels: app: your-app strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 25% template: metadata: labels: app: your-app spec: containers: - name: your-app-container image: your-docker-registry/your-app:latest # tag updates automatically ports: - containerPort: 3000 """ * **Don't Do This:** Deploy your application without a proper deployment strategy resulting in downtime. * **Why:** disrupts user experience and potentially causes loss of revenue. ### 2.8 Health Checks * **Do This:** Implement health check endpoints for your application. * **Why:** Allows load balancers and orchestration tools (like Kubernetes) to monitor the health of your application instances. """crystal require "kemal" get "/health" do |context| context.response.status_code = 200 context.response.print "OK" end Kemal.run """ * **Don't Do This:** Deploy without a health check. * **Why:** Makes automatic recovery and scaling difficult. ## 3. Performance Optimization ### 3.1 General Principles * **Profiling:** Identify performance bottlenecks. * **Caching:** Implement caching strategies. * **Resource Limits:** Define memory and CPU limits. ### 3.2 Profiling * **Do This:** Use tools like "perf" or "flamegraph" to profile your application. * **Why:** Helps you identify and address performance bottlenecks. """bash # Example profiling with perf perf record -g ./your_app perf report """ * **Don't Do This:** Make performance optimizations without profiling first. * **Why:** Wastes time and effort on potentially insignificant issues. ### 3.3 Caching * **Do This:** Implement caching using solutions like Redis or Memcached. * **Why:** Reduces database load and improves response times. """crystal require "redis" redis = Redis.new(host: "localhost", port: 6379) def get_data(key : String) cached_data = redis.get(key) if cached_data return cached_data else # Fetch data from the database data = fetch_from_db(key) redis.setex(key, 3600, data) # Cache for one hour return data end end """ * **Don't Do This:** Cache data indefinitely without invalidation. * **Why:** Can lead to stale data and incorrect results. ### 3.4 Resource Limits * **Do This:** Set memory and CPU limits for your containers or processes. * **Why:** Prevents resource exhaustion and improves stability. """yaml # Kubernetes Pod example with resource limits apiVersion: v1 kind: Pod metadata: name: your-app-pod spec: containers: - name: your-app-container image: your-docker-registry/your-app:latest resources: limits: memory: "512Mi" cpu: "1" requests: memory: "256Mi" cpu: "0.5" """ * **Don't Do This:** Fail to set resource limits. * **Why:** Increases the risk of resource depletion and application crashes. ### 3.5 Cluster Management * **Do This:** Create a cluster using Kubernetes, ECS, or similar solutions. * **Why:** Allows you to scale your app quickly, and reliably. """yaml # Sample Kubernetes Deployment manifest for scaling apiVersion: apps/v1 kind: Deployment metadata: name: crystal-app spec: replicas: 3 # Start with 3 replicas selector: matchLabels: app: crystal-app template: metadata: labels: app: crystal-app spec: containers: - name: crystal-app image: your-repo/crystal-app:latest ports: - containerPort: 8080 """ **Explanation:** This Deployment manifest creates three replicas, allowing you to distribute traffic. You could scale up or down by altering the replication factor. * **Don't Do This:** Have a single point of failure for your application. * **Why:** A cluster ensures higher availability and better scalability. ## 4. Database Management ### 4.1 General Principles * **Connection Pooling:** Reuse database connections. * **Migrations:** Manage database schema changes. * **Backups:** Implement regular database backups. ### 4.2 Connection Pooling * **Do This:** Use a connection pool to manage database connections. * **Why:** Reduces the overhead of establishing new connections for each request. """crystal require "pg" pool = PG::Pool.new(host: "localhost", port: 5432, user: "username", password: "password", database: "mydb", max_connections: 10) def query_db(sql : String) pool.use do |connection| result = connection.exec(sql) # Process the result end end """ * **Don't Do This:** Create a new database connection for each request. * **Why:** Inefficient and can exhaust database resources. ### 4.3 Migrations * **Do This:** Use a migration tool like "migrate" or implement your own migration system. * **Why:** Allows you to manage database schema changes in a controlled and versioned manner. """crystal # Example migration using a hypothetical migration library require "my_migration_lib" migration = MyMigrationLib::Migration.new migration.up do execute("CREATE TABLE users (id SERIAL PRIMARY KEY, name VARCHAR(255))") end migration.down do execute("DROP TABLE users") end """ * **Don't Do This:** Manually alter the database schema without a migration system. * **Why:** Difficult to track changes and revert if necessary. ### 4.4 Backups * **Do This:** Implement regular database backups. * **Why:** Protects your data from loss in case of hardware failures or other disasters. """bash # Example database backup script pg_dump -h localhost -p 5432 -U username -d mydb -f backup.sql """ * **Don't Do This:** Neglect to back up your database. * **Why:** Could lead to permanent data loss. ## 5. Conclusion Adhering to these DevOps standards will help you build robust, scalable, and maintainable Crystal applications. By automating build processes, implementing comprehensive logging and monitoring, and following security best practices, you can ensure the reliability and performance of your applications in production. Continue to iterate and refine these standards as Crystal evolves and new best practices emerge.