pyvider-hcl Architecture¶
This document provides a visual and detailed overview of the pyvider-hcl architecture.
🤖 AI-Generated Content
This documentation was generated with AI assistance and is still being audited. Some, or potentially a lot, of this information may be inaccurate. Learn more.
System Overview¶
┌─────────────────────────────────────────────────────────────────┐
│ User Application │
└───────────────────────────────┬─────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ pyvider.hcl Package │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌────────────────┐│
│ │ parser/ │ │ factories/ │ │ output/ ││
│ │ ├─ base.py │ │ ├─ types.py │ │ └─ formatting ││
│ │ ├─ inference.py│ │ ├─ variables.py │ │ .py ││
│ │ └─ context.py │ │ └─ resources.py │ │ ││
│ │ │ │ │ │ ││
│ │ • parse_hcl_ │ │ • create_ │ │ • pretty_print ││
│ │ to_cty() │ │ variable_cty │ │ _cty() ││
│ │ • parse_with_ │ │ • create_ │ │ ││
│ │ context() │ │ resource_cty │ │ ││
│ │ • auto_infer_ │ │ • parse_hcl_ │ │ ││
│ │ cty_type() │ │ type_string() │ │ ││
│ └─────────────────┘ └──────────────────┘ └────────────────┘│
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ terraform/ │ │ exceptions.py │ │
│ │ └─ config.py │ │ • HclError │ │
│ │ │ │ • HclParsing │ │
│ │ • parse_ │ │ Error │ │
│ │ terraform_ │ │ │ │
│ │ config() │ │ │ │
│ │ (placeholder) │ │ │ │
│ └──────────────────┘ └──────────────────┘ │
└───────────────────────────────┬─────────────────────────────────┘
│
┌───────────────┼───────────────┐
│ │ │
▼ ▼ ▼
┌──────────────────┐ ┌──────────────┐ ┌──────────────────┐
│ python-hcl2 │ │ pyvider-cty │ │ provide- │
│ (HCL parser) │ │ (type system)│ │ foundation │
│ │ │ │ │ (errors/logging) │
└──────────────────┘ └──────────────┘ └──────────────────┘
Module Breakdown¶
1. Parser Subpackage (parser/)¶
Modules:
- base.py: Core parsing logic - contains parse_hcl_to_cty()
- inference.py: Type inference - contains auto_infer_cty_type()
- context.py: Enhanced error context - contains parse_with_context()
Responsibilities: - Parse HCL strings into Python data structures - Convert Python data to CTY types - Automatic type inference - Schema validation
Data Flow:
HCL String
↓
parse_hcl_to_cty() [base.py]
↓
python-hcl2.loads() ──→ Python dict/list
↓
Schema provided?
├─ Yes → schema.validate(data) ──→ CtyValue
└─ No → auto_infer_cty_type() [inference.py] ──→ CtyValue
Key Functions:
- parse_hcl_to_cty(hcl_content, schema=None) → CtyValue (base.py)
- parse_with_context(content, source_file=None) → dict (context.py)
- auto_infer_cty_type(raw_data) → CtyValue (inference.py)
2. Factories Subpackage (factories/)¶
Modules:
- types.py: Type string parsing - contains parse_hcl_type_string()
- variables.py: Variable factory - contains create_variable_cty()
- resources.py: Resource factory - contains create_resource_cty()
Responsibilities: - Create Terraform variable structures - Create Terraform resource structures - Parse HCL type strings into CTY types
Type String Parsing Flow:
Type String (e.g., "list(string)")
↓
parse_hcl_type_string() [types.py]
├─ Primitive? → Return CtyString/CtyNumber/CtyBool
├─ list(T)? → Return CtyList(element_type=T)
├─ map(T)? → Return CtyMap(element_type=T)
└─ object({})?→ Parse attributes → Return CtyObject
Variable Creation Flow:
Python inputs
↓
create_variable_cty() [variables.py]
↓
Parse type_str → CtyType [types.py]
↓
Validate default against type
↓
Build Terraform variable structure
↓
Validate with schema → CtyValue
Key Functions:
- create_variable_cty(name, type_str, default_py=None, ...) → CtyValue (variables.py)
- create_resource_cty(r_type, r_name, attributes_py, ...) → CtyValue (resources.py)
- parse_hcl_type_string(type_str) → CtyType (types.py)
3. Output Subpackage (output/)¶
Modules:
- formatting.py: CTY value formatting - contains pretty_print_cty()
Responsibilities: - Format CTY values for human-readable display - Handle nested structures (objects, lists, maps, tuples)
Printing Flow:
CtyValue
↓
pretty_print_cty() [formatting.py]
↓
_pretty_print_cty_recursive()
├─ CtyObject → Format as JSON object
├─ CtyList → Format as JSON array
├─ CtyMap → Format as JSON object
├─ CtyTuple → Format as JSON array
└─ Primitive → Format as string/number/bool
Key Functions:
- pretty_print_cty(value) → None (prints to stdout) (formatting.py)
4. Terraform Subpackage (terraform/)¶
Modules:
- config.py: Terraform configuration parsing - contains parse_terraform_config()
Status: Placeholder for future implementation
Planned Responsibilities: - Parse Terraform-specific blocks (provider, module, data, etc.) - Handle Terraform configuration files - Validate Terraform-specific structures
Current:
- parse_terraform_config(config_path) returns placeholder (config.py)
5. Exceptions Module (exceptions.py)¶
Responsibilities: - Define custom exception types - Provide structured error information - Integrate with provide-foundation error handling
Exception Hierarchy:
provide.foundation.FoundationError
↓
HclError (base class)
↓
HclParsingError
├─ message: str
├─ source_file: str | None
├─ line: int | None
└─ column: int | None
Data Flow Examples¶
Example 1: Parse HCL with Schema¶
User Code:
hcl_string = 'name = "test"'
schema = CtyObject({"name": CtyString()})
result = parse_hcl_to_cty(hcl_string, schema)
Flow:
1. parse_hcl_to_cty() receives string and schema
2. python-hcl2.loads() parses HCL → {"name": "test"}
3. schema.validate({"name": "test"}) validates
4. Returns CtyValue with type=CtyObject, value={"name": CtyString("test")}
Example 2: Create Terraform Variable¶
User Code:
var = create_variable_cty(
name="region",
type_str="string",
default_py="us-west-2"
)
Flow:
1. create_variable_cty() receives params
2. _parse_hcl_type_string("string") → CtyString()
3. CtyString().validate("us-west-2") → validates
4. Build structure: {"variable": [{"region": {...}}]}
5. Create schema for validation
6. Return validated CtyValue
Example 3: Automatic Type Inference¶
User Code:
hcl = '''
name = "example"
count = 5
enabled = true
'''
result = parse_hcl_to_cty(hcl)
Flow:
1. parse_hcl_to_cty() with no schema
2. python-hcl2.loads() → {"name": "example", "count": 5, "enabled": true}
3. auto_infer_cty_type() walks the data:
- "example" → CtyString
- 5 → CtyNumber
- true → CtyBool
4. Build CtyObject with inferred types
5. Return CtyValue
Type System Integration¶
CTY Type Mapping¶
HCL Type String → CTY Type
─────────────────────────────────────
"string" → CtyString()
"number" → CtyNumber()
"bool" → CtyBool()
"any" → CtyDynamic()
"list(string)" → CtyList(element_type=CtyString())
"map(number)" → CtyMap(element_type=CtyNumber())
"object({...})" → CtyObject(attributes={...})
Python to CTY Inference¶
Python Value → CTY Type
─────────────────────────────────────
str → CtyString()
int/float/Decimal → CtyNumber()
bool → CtyBool()
None → CtyDynamic()
list → CtyList(CtyDynamic())
dict → CtyObject({...})
Error Handling Flow¶
Error Occurs
↓
Which layer?
├─ HCL Parsing → python-hcl2 exception
│ ↓
│ Caught by parse_hcl_to_cty()
│ ↓
│ Wrapped in HclParsingError
│
├─ Schema Validation → CtyValidationError
│ ↓
│ Caught by parse_hcl_to_cty()
│ ↓
│ Wrapped in HclParsingError
│
└─ Factory → HclFactoryError
↓
Raised directly
↓
User catches exception with:
- Descriptive message
- Source location (if available)
- Original error context
Performance Characteristics¶
Current Implementation: - Parsing: O(n) where n = HCL content size (via python-hcl2) - Type Inference: O(m) where m = number of fields in data structure - Schema Validation: O(m) for field validation - No caching: Each parse is independent - No lazy evaluation: All parsing happens immediately
Memory Usage: - HCL string kept in memory - Full parse tree created in memory - CTY objects created for all values - Typical: ~2-5x HCL string size
Extension Points¶
To Add New Features:¶
- New Parser Functions:
- Add to appropriate module in
parser/subpackage - Export in
__init__.py -
Add tests in
tests/parser/test_parser.py -
New Factory Types:
- Add factory function to appropriate module in
factories/subpackage - Export in
__init__.py -
Add tests in
tests/factories/test_factories.py -
New Type Support:
- Extend
parse_hcl_type_string()infactories/types.py - Update
PRIMITIVE_TYPE_MAPorCOMPLEX_TYPE_REGEX -
Add corresponding CTY types from pyvider-cty
-
Terraform Features:
- Implement in appropriate module in
terraform/subpackage - May need new parsing logic
- Will integrate with existing parser/factory modules
Dependencies¶
Runtime Dependencies:
- python-hcl2>=7.2.1 - Core HCL parsing
- pyvider-cty>=0.0.113 - Type system
- provide-foundation>=0.0.0 - Error handling/logging
- attrs>=25.3.0 - Structured exceptions
- regex>=2024.11.6 - Enhanced regex
Development Dependencies:
- pytest - Testing framework
- pytest-xdist - Parallel test execution
- ruff - Linting and formatting
- mypy - Type checking
- provide-testkit[standard,advanced-testing,typecheck,build] - Test utilities
Design Principles¶
- Simplicity: Focused API with minimal abstractions
- Type Safety: All values go through CTY type system
- Error Context: Rich error messages with source locations
- Composability: Small, focused modules that work together
- Extensibility: Clear extension points for future features
- Testability: All functions are pure and easily testable